#malli

generated UTC: 2023-02-13 19:06
latest data: https://clojurians-log.clojureverse.org/malli/2023-02-12
messages: 9000
pro tips:
* Double click on text to filter by it. (doubleclick + cmd-f for extra points).
* Click on date to keep day visible regardless of filter.
* Click on time to keep hour visible regardless of filter.
#2019-07-1414:39ikitommiAny comments and critique on design welcome#2019-08-0115:24ikitommi
(defn composite-perf2 []
  (let [tests [[true [-1]]
               [true [-1 1 2]]
               [false [-1 0 2]]
               [false [-1 -1 -1 -1]]]
        assert! (fn [f]
                  (doseq [[expected data] tests]
                    (assert (= expected (f data)))))]

    ;; 155ns
    (let [valid? (fn [x]
                   (and (vector? x)
                        (<= (count x) 3)
                        (every? #(and (int? %) (or (pos-int? %) (neg-int? %))) x)))]
      (assert! valid?)
      (cc/quick-bench
        (valid? [-1 1 2])))

    ;; 27ns
    (let [valid? (m/validator
                   [:vector {:max 3}
                    [:and int? [:or pos-int? neg-int?]]])]
      (assert! valid?)
      (cc/quick-bench
        (valid? [-1 1 2])))

    ;; 506ns
    (let [spec (s/coll-of
                 (s/and int? (s/or :pos-int pos-int? :neg-int neg-int?))
                 :kind vector?
                 :max-count 3)
          valid? (partial s/valid? spec)]
      (assert! valid?)
      (cc/quick-bench
        (valid? [-1 1 2])))))
#2019-08-0209:02ikitommisome discussion about defaults: https://github.com/metosin/malli/issues/25#2019-10-0319:17bedershey there, I was trying to play with malli a bit today#2019-10-0319:17bedershaving trouble downloading its dependencies though#2019-10-0319:17bedersI used the latest commit id and `{:git/url "https://github.com/metosin/malli" :sha "59b226968025502e3830f5c6404b0f95f8c080e6"}`#2019-10-0319:18bedersNot super familiar with deps I must admit#2019-10-0319:18bedersgetting this error at the moment:
Error building classpath. Unable to compare versions for borkdude/edamame: {:mvn/version "0.0.5-SNAPSHOT", :deps/manifest :mvn} and {:git/url "", :sha "b577e565b136d3dd51945fe874049d4297946f57", :deps/manifest :deps, :deps/root "/Users/beders/.gitlibs/libs/borkdude/edamame/b577e565b136d3dd51945fe874049d4297946f57"}
#2019-10-0319:45ikitommi@beders hmm.. not sure what’s wrong with edamame, but this should work:
clj -Sdeps '{:deps {metosin/malli {:git/url "" :sha "10b09bef72a52755764ba21933edc983fc4135e7"}, borkdude/edamame {:git/url "", :sha "d7d0f12c336dc513702c8c704b147223f85e377f"}}}'
#2019-10-0319:45bedersthank you!#2019-10-0511:07ikitommiasked in #tools-deps how to resolve that, meanwhile, changed edamame to use mvn versio, so this works now:
clj -Sdeps '{:deps {metosin/malli {:git/url "" :sha "22122b2ff9434bfd0db8d72fcd06cb48914c6eb4"}}}'
#2019-10-0818:49sammikkohi, couldnt find channel for jsonista.. is it possible to de-serialize json date strings "2019-10-04T09:08:36.078Z" to org.joda.time.DateTime objects, based on the format of the incoming string... i understand it works the other way around#2019-10-0818:52sammikkoi'd just like to get a clojure map from json string, but the strings looking like dates should be converted into DateTime..#2019-10-0820:09ikitommiyou need to describe which fields should be dates and apply coercion with that. E.g. need to use spec/schema or malli.#2019-10-0820:10ikitommibased on format... you could also roll your own walker that inspects the strings, but would be slow I think#2019-10-0907:28sammikkoyeah.. we are using Schema atm and schema bijections library to do the conversions.. but its very slow..#2019-10-0907:38ikitommiSchema coercion should be fast? Malli will be Really Fast, but doesn't have an easy way to create custom transforming schemas. One PR away from....#2019-10-0907:58sammikkothanks.. I'll check out coercion as well..#2019-10-0908:26ikitommiok, could be faster. the date parsing on java-side seems quite slow in general:
(require '[malli.core :as m])
(require '[malli.transform :as mt])

(def Pizza
  [:map
   [:id int?]
   [:name string?]
   [:created inst?]])

(def json->Pizza
  (m/transformer
    Pizza
    mt/json-transformer))


(json->Pizza
  {:id 1
   :name "salami"
   :created "2019-10-04T09:08:36.078Z"})
;{:id 1
; :name "salami"
; :created #inst"2019-10-04T09:08:36.078-00:00"}

(require '[criterium.core :as cc])

(cc/quick-bench
  (json->Pizza
    {:id 1
     :name "salami"
     :created "2019-10-04T09:08:36.078Z"}))
; Evaluation count : 386256 in 6 samples of 64376 calls.
;              Execution time mean : 1.719360 µs
;     Execution time std-deviation : 100.090402 ns
;    Execution time lower quantile : 1.601860 µs ( 2.5%)
;    Execution time upper quantile : 1.811632 µs (97.5%)
;                    Overhead used : 1.862824 ns
#2019-10-0908:29ikitommioh, Plumatic Schema doesn’t have date coercion oob, forgot about that.#2019-10-0908:32ikitommiwith schema-tools:
(require '[schema.core :as s])
(require '[schema-tools.coerce :as stc])

(def json->Pizza2
  (stc/coercer {:id s/Int, :name s/Str, :created s/Inst} stc/json-coercion-matcher))

(json->Pizza2
  {:id 1
   :name "salami"
   :created "2019-10-04T09:08:36.078Z"})
;{:id 1
; :name "salami"
; :created #inst"2019-10-04T09:08:36.078-00:00"}

(cc/quick-bench
  (json->Pizza2
    {:id 1
     :name "salami"
     :created "2019-10-04T09:08:36.078Z"}))
; Evaluation count : 261396 in 6 samples of 43566 calls.
;              Execution time mean : 2.352737 µs
;     Execution time std-deviation : 173.369968 ns
;    Execution time lower quantile : 2.255370 µs ( 2.5%)
;    Execution time upper quantile : 2.636725 µs (97.5%)
;                    Overhead used : 1.600826 ns
#2019-10-2517:19sammikkohey @ikitommi what you think of specter lib.. i know you like performance etc., you guys like to use it? see any trouble with it?#2019-10-2517:23sammikkoI get some 90s flashbacks of the XSLT and xpath times, but it sure seems convenient..#2019-10-2813:06eskosHey so I feel really dumb but how would I make a malli schema which validates that I have a non-empty string 😅 I’m thinking something like [:and string? [:not= blank?]] but there’s no blank? predicate support…#2019-10-2813:06eskosblank? does string check internally as well so it doesn’t need to be :and schema but anyhoos…#2019-10-2813:54ikitommi@suomi.esko You can use any predicate with :fn, either: [:fn str/blank?] (local, fast) or [:fn 'str/blank?] (portable)#2019-10-2814:05eskos@ikitommi Seems to work for the positive case, but I really do want to test the negation of that and I can’t seem to get it to work, or at least :not= isn’t what I should use. I think there isn’t a complementing predicate yet? 🙂#2019-10-2814:07ikitommi[:and string? some?] or [:fn {:error/message "should be non-empty string"} '(complement str/blank?)]?#2019-10-2814:07ikitommicould be :not ....#2019-10-2814:09eskoslatter works though#2019-10-2814:11eskosformer indeed does not, some? is effectively “not nil” check#2019-10-3013:21eskos@ikitommi Had a thought, should I create an issue on GitHub about my “non-empty string” problem or do you think this is eventually solved as side effect or by other means?#2019-10-3014:03ikitommiissue welcome, and suggestion how to fix that 🙂#2019-10-3014:03ikitommihttps://malli.io/?value=%22%22&amp;schema=%5B%3Aand%20string%3F%20%5B%3Afn%20%7B%3Aerror%2Fmessage%20%22should%20not%20be%20empty%22%7D%20(complement%20str%2Fblank%3F)%5D%5D#2019-10-3014:07Toni VanhalaThere’s not-empty in core, while blank? is in clojure.string#2019-10-3014:08Toni VanhalaI would expect [:and string? not-empty] to work out-of-the-box, while blank? is arguable#2019-10-3014:10ikitomminot-empty is not registered by default, should it be?#2019-10-3014:11ikitommi
[:and string? [:fn {:error/message "should not be empty"} 'not-empty]]
#2019-10-3014:11ikitommiworks#2019-10-3014:11ikitommihttps://malli.io/?value=%221%22&amp;schema=%5B%3Aand%0A%20string%3F%0A%20%5B%3Afn%20%7B%3Aerror%2Fmessage%20%22should%20not%20be%20empty%22%7D%20not-empty%5D%5D#2019-10-3014:15eskosWould be nice 🙂#2019-10-3014:16eskosAlthough I think complements are also useful in general…#2019-10-3109:47orestisI'm evaluating various "spec" libraries for use in production. It's a bit of a sticky situation, since clojure.spec is alpha, clojure.spec2 and malli are pre-alpha. The only "stable" library is Plumatic Schema, but that's also old (though actively maintained).#2019-10-3109:48orestisBut, everyone's definition of pre-alpha is different. Alex Miller explicitly says "don't use spec2 in production yet, eta is in the order of months".#2019-10-3109:48orestisWhat is the definition of pre-alpha for Malli?#2019-10-3109:57ikitommiwe wanted to optimize the whole - currently all non-core modules (generators, providers, transformers, etc) have initial versions and we know how to core should be done to support the needs of the modules. There will be few namespace & function renames (e.g. breaking) after which we’ll put a alpha out. Will be in alpha as long all the relevant features are feature-complete, but I would guess there will be a real release within 1-2 months.#2019-10-3109:59ikitommithere is for example now m/transform function but it will be split into m/encode and m/decode, like in spec-tools. Didn’t want to start maintaining CHANGELOG for all the things that are in a flux now, but final design starts to emerge, so getting close.#2019-10-3109:59ikitommialso, there are some big things that need to be decided before release, will write a post &/ poll out of those.#2019-10-3110:00ikitommi• open or closed maps by default? (open) • support clojure core predicates (`ìnt?`, string?) by default or make keyword type schemas (`:int`, :string, ...)#2019-10-3110:02ikitommiwe’ll push a reitit coercion module of malli soon, so people can start playing (and reporting issues) with it.#2019-10-3113:18orestisThanks for the update! Looking forward to see what's cooking.#2019-11-0107:34ikitommistill surprised how well the schema/hiccup syntax supports elegant extensions: https://github.com/metosin/malli/issues/105#2019-11-0310:06eskosI'd claim that's the benefit of supple systems 🙂 Not only hiccup syntax has low mental load, it is easy to reason without switching contexts just for the specific task and being just data it can be rustled as lightly or heavily as one needs to... TBH this is my primary reason I'm now lurking here, using hiccup syntax is definitely if not the absolutely correct, at least a very good idea as it is very approachable.#2019-11-0218:10roklenarcicI have a question related to malli, but it's probably just an error on my part but I cannot spot it: I defined macro as this (simplified example):
(defmacro x [y]
  `(println ~(m/schema [:tuple y])))
And when I call (x int?) I get:
Syntax error compiling fn* at (/private/var/folders/fm/5mzhclpd7mj0tzjjq796ftc00000gn/T/form-init14721255528686671623.clj:1:1).
Can't embed object in code, maybe print-dup not defined: 
However if I define macro as this it works:
(defmacro x [y]
  `(println (m/schema [:tuple ~y])))
=> #'clj-rest-client.core/x
(x int?)
[:tuple int?]
#2019-11-0218:11roklenarcicI wanted to compute the schema value at macro expansion time to make it faster, but it doesn't seem to work for some reason#2019-11-0218:11roklenarcicmaybe someone else can spot it, I've been looking at this for a while with no idea#2019-11-0218:38ikitommi@roklenarcic macros should return source code, the first example returns a reified protocol instance. You can run (macroexpand ...) to that#2019-11-0218:39ikitommiJust curious, "to make it faster"?#2019-11-0218:39roklenarcicI thought that m/schema does some preprocessing#2019-11-0218:39roklenarcicwhich would make it faster than sticking simply [:tuple x] in there#2019-11-0218:40roklenarcicbtw, it seems that [:tuple] doesn;'t work#2019-11-0218:40roklenarcic(m/schema [:tuple]) -> :tuple but (m/schema [:tuple int?]) -> [:tuple int?]#2019-11-0218:41ikitommimalli has it's own compiler for validation, explain & transform. Reaults of those should be pretty fast.#2019-11-0218:42ikitommiCreation of schema has some overhead, but if you can store a reference to a transformer, it should be as fast as one can do with Clojure.#2019-11-0218:43roklenarcicthanks for the info#2019-11-0218:46ikitommithe :tuple thing... some schemas has a constraint that they should have 1+ childs. I guess tuple doesn't and empty tuple is ok...#2019-11-0218:46roklenarcicI had spec1 code that said -> give me a vector of values and a vector of specs and I will generate a s/fdef spec (which uses s/cat), so the code generated a spec like this (s/cat :arg1 arg-spec1 ...) and potentially (s/cat) which took an empty sequence just fine. I replaced this with :tuple from malli and it seems to have a corner case for empty tuple#2019-11-0218:47ikitommishould it require at least 1 child?#2019-11-0218:47roklenarcicit works the opposite#2019-11-0218:48roklenarcic
(m/explain [:tuple] [])
Execution error (NullPointerException) at malli.core/-tuple-schema$reify$reify$fn (core.cljc:447).
null
#2019-11-0218:48roklenarcicbut spec with empty (s/cat) would work fine#2019-11-0218:48ikitommiOh, that's a bug.#2019-11-0218:48roklenarcicyeah I thought so#2019-11-0218:49roklenarcicNPEs are usually not intended#2019-11-0218:49roklenarcicI'm porting my rest client lib from spec to malli now... hopefully I'll have encode/decode soon#2019-11-0218:53roklenarcicI'd like for user to be able to specify the transformation to string I should do before sending any opaque types over the wire#2019-11-0218:53roklenarcictransforms seem to be doing the opposite direction#2019-11-0218:55roklenarcice.g. if user has LocalDateTime object I'd like for them to specify format in schema that I format it to before I send it...#2019-11-0219:01ikitommihttps://github.com/metosin/malli/commit/c2f6b45bd12751f18b737cb227d17fae7f024663#2019-11-0219:04ikitommiencode & decode is done (#99), but requires (#98) before will merge that in.#2019-11-0219:16roklenarcicI'll be finally rid of the nightmare that is trying to conform s/merge specs#2019-11-0300:44roklenarcicBase Registry includes schemas :vector, :list, :set, but what I'm missing here is :sequential. Often I have use cases where I don't particularly care which collection it is as long as I can process it like a sequence.#2019-11-0311:50roklenarcicHow would I write (s/coll-of string?) in malli?#2019-11-0313:01ikitommiI think :sequential would be nice. Is :list actually useful at all?#2019-11-0313:18roklenarcicI mean it is if you want to schema a list but thats so rare except when you’re validating macro arguments#2019-11-0313:19roklenarcicThe operators i miss the most are coll-of and every#2019-11-0412:50ikitommi@roklenarcic in latest master, you can use IntoSchema values as element in the schema ast:
(require '[malli.core :as m])

(def sequential (#'m/-collection-schema `sequential sequential? seq nil))

(m/validate [sequential int?] '(1 2 3)) 
; => true

(m/validate [sequential int?] [1 2 3]) 
; => true

(m/form [sequential int?])
; => [user/sequential int?]
#2019-11-0412:54ikitommihttps://github.com/metosin/malli/commit/adc9994043977965cdca3e37bfa3187c36cc6b1b#2019-11-0414:57roklenarcicnice trick to get around private var 🙂#2019-11-0417:24ikitomminot complete, transforming needs to retain the original type, but an example of all the extensions implemented and tested for a schema: https://github.com/metosin/malli/pull/110#2019-11-0417:25ikitommiI'm thinking of adding a supporting protocol for the different extension, so one could just implement all the concerns with one reify.#2019-11-0417:26ikitommimalli.json-schema/JsonSchema etc.#2019-11-0417:31ikitomminot going to use the protocols in the core, e.g. core doesn't depend on anything to keep it small, but for custom new client schemas, I think the all-protocols approach is more declarative than the mixture of protocols and mms#2019-11-0511:38plexusSeems there's an issue when using a custom registry with generate. The generators don't pass the opts down into m/schema, so when you have nested schemas they end up using the default registry#2019-11-0511:42plexusor I'm holding it wrong 🙂 trying to come up with a minimal repro#2019-11-0512:10ikitommihmm… quick look says the sequential schemas are not passing the opts. PR welcome @plexus#2019-11-0512:11plexushttps://github.com/metosin/malli/pull/112#2019-11-0512:12ikitommithanks, merged#2019-11-0512:12plexussweet!#2019-11-0512:13plexusside note, is it too late to change childs to children?#2019-11-0512:13plexusit bothers me every time 🙂#2019-11-0512:13ikitommi* children - 8 700 000 000 hits * childs - 75 200 000 hits#2019-11-0512:14ikitommiit’s pre-alpha, good time for fixing things like this. Do you have time for a PR?#2019-11-0512:15ikitommi> The difficulty with the plural began in Old English, where the nominative plural was at first cild, identical with the singular, then c.975 a plural form cildru (genitive cildra) arose, probably for clarity’s sake, only to be re-pluraled late 12c. as children, which is thus a double plural. Middle English plural cildre survives in Lancashire dialect childer and in Childermas.#2019-11-0512:15plexusinteresting! wiktionary is a bit more harsh#2019-11-0512:16plexus#2019-11-0512:16plexusyeah, I can do a PR for that later today#2019-11-0512:30plexusI see I also broke some tests... will send a PR for that as well#2019-11-0513:45plexusnew PR sent. I really appreciate that the project has a bin/kaocha, it's so great not to have to think about how to run the tests on each project.#2019-11-0514:00ikitommithanks for the PR kaocha, makes things kinda simple 🙂#2019-11-1217:15ikitommitwo new PRs in master: support for two-way transformations (`m/decode` & m/encode), :sequential schema and allowing any schema properties based (serializable!) encoders & decoders#2019-11-1217:21ikitommi
(let [schema (-> [:and {:title "lower-upper-string"
                        :decode/string '(constantly str/upper-case)
                        :encode/string '(constantly str/lower-case)}
                  string?]
                 (malli.edn/write-string)
                 (malli.edn/read-string))]
  (as-> "kikka" $
        (m/decode schema $ mt/string-transformer)
        (doto $ prn)
        (m/encode schema $ mt/string-transformer)
        (doto $ prn)
        schema))
; prints "KIKKA"
; prints "kikka"
; => [:and
;     {:title "lower-upper-string"
;      :decode/string (constantly str/upper-case)
;      :encode/string (constantly str/lower-case)}
;      string?]
#2019-11-1217:26ikitommiThe two-way thing looks from outside the same as with spec-tools, but it actually works 🙂#2019-11-1217:27ikitommi… thanks to clear separation of transformation and validation. the encode mostly creates values that are not valid (e.g. a string representation of a date), and had to add all kind of hacks to make that work with clojure.spec.#2019-11-1406:20eskosI’m not sure if I like that str is directly usable in schemas like that. From user point of view I’d assume my own namespace requires to apply, not implicit ones from within malli and in that case I’d rather see an error…I do see the usefulness of that, but it makes me go a bit :face_with_raised_eyebrow: Eh, maybe that’s just me…#2019-11-1406:45ikitommiyou can explicitly allow certain qualified symbols to be used, could allow that via global options. I guess clojure.string is a dependency and is available for users of malli. BUT, might fail with advanced compilation unless explixitely registered those functions. Explicit is always better…#2019-11-1406:48ikitommithough of exposing explicitely some clojure.test.check.generators functions for generators, to enable portable generators with :gen/gen schema property. Those definitions should be in optional ns so if not needed, doesn’t make the bundle size bigger.#2019-11-1406:49ikitommibetter portable example in a tweet: https://twitter.com/ikitommi/status/1194555187995848705#2019-11-1406:51ikitommi@suomi.esko actually, the str stuff is explicitly allowed in sci, so ok: https://github.com/borkdude/sci/blob/9b70708c5c42e89d94a72aa961bfa1d3a6ef8be4/src/sci/impl/namespaces.cljc#L437-L475#2019-11-1406:52eskos¯\(ツ)/¯#2019-11-1518:34pithylessWhat is the expected way to check for a function? Something like [:map [:handler fn?]] It looks like fn? is not one of the symbols exported in https://github.com/borkdude/sci/blob/cb1dc3139a53c680b2bdc1038d2c6e26b975ee8e/src/sci/impl/namespaces.cljc#L96 I could do something like [:map [:handler [:fn fn?]]], but I'm wondering if this is something that will be supported for serialization.#2019-11-1519:24ikitommiI think @borkdude knows if sci could support fn?#2019-11-1519:25borkdudesure. PR welcome, it should be added about here: https://github.com/borkdude/sci/blob/ab5ba82fa61ab47b7ace8b3e58bfa633ae8e1f72/src/sci/impl/namespaces.cljc#L201#2019-11-1519:25borkdudeslipped through, don't know why#2019-11-1519:26ikitommiAwesome, [:map [:handler [:fn 'fn?]]] would work after that & survive the serialization#2019-11-1622:32borkdudeI just added fn? to sci master#2019-11-1806:13danielgrosseHi, when adding the latest ref from malli in the deps.edn and starting a repl, I've got this error:
error in process sentinel: Could not start nREPL server: Error building classpath. Unable to compare versions for borkdude/edamame: {:mvn/version "0.0.8-alpha.4", :deps/manifest :mvn} and {:git/url "", :sha "34755e7a875fae2ab4d64e6be89b6270046a91a3", :deps/manifest :deps, :deps/root "/Users/xxx/.gitlibs/libs/borkdude/edamame/34755e7a875fae2ab4d64e6be89b6270046a91a3"}
#2019-11-1806:14danielgrosseAny tips how I could solve this?#2019-11-1806:21ikitommi@danielgrosse fixed that on master - for edamame, it uses now the “0.0.8-alpha.4” version. DEPS doesn’t know which one is latest that or the SHA (which was later).#2019-11-1806:21ikitommi
➜  ~ clj -Sdeps '{:deps {metosin/malli {:git/url ""
                                        :sha "e2ff736ff923c676de655b853180be2503715d88"}}}'
Checking out:  at e2ff736ff923c676de655b853180be2503715d88
Clojure 1.10.1
user=>
#2019-11-1806:22ikitommiNot sure if that is a transitive dependency bug of deps, or just a feature.#2019-11-1806:23ikitommi@pithyless master uses the latest version of sci, so [:map [:handler [:fn 'fn?]]] should work now.#2019-11-1806:24danielgrosseThanks for the fast fix. :thumbsup:#2019-11-1806:31ikitommiIt was an easy fix 😉#2019-11-1806:32ikitommibtw my next steps are to finish :multi schema and the pretty error printer btw, both 80% done.#2019-11-1806:32danielgrosseCool. Keep up the great work.#2019-11-1806:33ikitommirecursive schemas & support for local registries might come via this PR https://github.com/metosin/malli/pull/117#issuecomment-554737541#2019-11-1809:40eskosTangential thought on documentation, would absolutely love to see documentation like this for malli eventually: https://pomb.us/build-your-own-react/ scroll a bit to see what I mean - the library behind that is https://github.com/pomber/code-surfer/blob/code-surfer-v2/readme.md#2019-11-1812:53ikitommi@suomi.esko feel free to build the docs. looks good, despite not being interactive.#2019-11-1917:23eskosI’ll try to give this a swing when I get some free time just to see if the idea actually works.#2019-11-1917:23eskosGives me a good excuse to really dive into malli as well 🙂#2019-11-1918:10ikitommilooking forward to seeing the new docs ;)#2019-11-2006:58eskosthey might end up being trash ¯\(ツ)/¯ but then we’ll know why#2019-11-1914:00ikitommiOh, the :multi popped out from the train: https://github.com/metosin/malli/pull/118#2019-11-2010:28kszaboAre there any plans of supporting clojure.spec1/2? The value proposition of Malli looks nice, but tooling has moved towards supporting clojure.spec and to ease adoption an iterative approach would be nice, slowly redefining existing system clojure.specs to Malli based data-oriented schemas.#2019-11-2010:45miikkaWhat would supporting spec mean?#2019-11-2010:47miikkaI suppose you could try to compile Malli schemata to spec-tools data specs#2019-11-2010:47miikkaIn my experience, mixing clojure.spec and prismatic schema was not a big deal, though, so not sure there's big need for integration in practice.#2019-11-2010:51Toni Vanhala> I suppose you could try to compile Malli schemata to spec-tools data specs I think the requirement was other way around “redefining existing system clojure.specs to Malli”#2019-11-2010:57ikitommino activities currently for the interop with spec. Feel free to suggest new tooling for this.#2019-11-2011:33kszaboactually both directions would be great 🙂 thanks for the notice#2019-11-2106:17ikitommiAs schemas are responsible for parsing their own ast, we could have container schemas for doing translation between modelling systems. Doesn’t help with spec as it’s built around the global registry & macros, but would work with declarative models like JSON Schema:
[:json-schema 
  {:type "object"
   :properties {:name {:type "string"}
                :number {:type "number"}
   :required [:name]}]
#2019-11-2211:13roklenarcicNot having a global registry has a downside though. Let’s say some library introduces a new Schema protocol implementation. Then someone uses that in schemas in namespace X. With Spec when you load the namespace you automatically gain all specs in there with all the required bits and pieces to make it all work. But with the way Malli is set up, if something is using a custom :datetime schema, then all the users of that thing must invoke all the Malli functions with an opts map where :registry is set to the default one with all the extensions added. If the calls to m/validate and such are burried inside a third party library, then that library must provide a way for you to pass the registry to be used otherwise, you cannot use any new schema types except the ones defined in malli.core#2019-11-2211:14roklenarcicAlso I’d like to ask, what is the policy regarding non-serializable content in schemas. I see that some examples include opts map where you have keys like {:decode/string some-fn} which isn’t really serializable#2019-11-2211:23roklenarcicThis is meant vis-a-vis developing own schema implementations… should I disallow non-serializable options?#2019-11-2212:09ikitommi@roklenarcic you can always define the library schemas as Vars, the first value in the Schema vector syntax can be an instance of IntoSchema.#2019-11-2212:10ikitommi
(def date-time (reify IntoSchema ...))

(m/validate date-time (java.time.DateTime.))
; => true
#2019-11-2212:11ikitommithe work like components on Reagent.#2019-11-2212:13roklenarcicso I can also do `[datetime “format”]?#2019-11-2212:13ikitommipolicy on non-serializable: up to the user. There is an issue about making an utility that checks if a schemas can be fully persisted or not. One can put that into project tests.#2019-11-2212:13ikitommiyes#2019-11-2212:13roklenarcicthanks that is very helpful#2019-11-2212:13roklenarcicI mean any schema that specifies decode/encode implementation cannot be persisted?#2019-11-2212:13ikitommi
(defprotocol IntoSchema
  (-into-schema [this properties children opts] "creates a new schema instance"))
#2019-11-2212:14ikitommiquote the functions and they can be serialized.#2019-11-2212:14ikitommi
(require '[malli.core :as m])
(require '[malli.edn :as edn])

(-> [:and
     [:map
      [:x int?]
      [:y int?]]
     [:fn '(fn [{:keys [x y]}] (> x y))]]
    (edn/write-string)
    (doto prn) ; => "[:and [:map [:x int?] [:y int?]] [:fn (fn [{:keys [x y]}] (> x y))]]"
    (edn/read-string)
    (doto (-> (m/validate {:x 0, :y 1}) prn)) ; => false
    (doto (-> (m/validate {:x 2, :y 1}) prn))) ; => true
;[:and 
; [:map 
;  [:x int?] 
;  [:y int?]] 
; [:fn (fn [{:keys [x y]}] (> x y))]]
#2019-11-2212:14roklenarcicah.. but then the code processing the value has to call eval or something on it?#2019-11-2212:15ikitommiit uses sci , a small interpreter. So it works on JVM, ClojureScript & GraalVM, no eval needed#2019-11-2212:15ikitommiit’s <10kb on cljs.#2019-11-2212:16ikitommihttps://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&amp;schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D#2019-11-2212:16ikitommithat’s all normal code, no clojurescript compiler used in the demo.#2019-11-2212:17roklenarcicsorry, I should be clearer. If my custom datetime schema has an option :formatter which someone puts the value of DateTimeFormatter/ISO_INSTANT#2019-11-2212:17ikitommie.g. “production code”#2019-11-2212:17ikitommioh, that. can’t serialize that.#2019-11-2212:17roklenarcicif I quote this, I get a symbol#2019-11-2212:17ikitommiit could have :format option, which could be string?#2019-11-2212:18borkdudethe latest version of sci supports Java reflection, if you bring this class in with the :classes option. but that won't work in CLJS of course#2019-11-2212:18roklenarcicsure, but you cannot specify instant formatter by giving a string#2019-11-2212:18borkdudebut you can use reader conditionals for that maybe#2019-11-2212:18roklenarcicspecifying something like “yyyy-MM-dd” then telling it to format an Instant object will fail#2019-11-2212:19ikitommi{:format :iso_instant} maybe?#2019-11-2212:19ikitommitime is hard to get do in a portable way anyway.#2019-11-2212:19roklenarcicMy current idea is to support everything#2019-11-2212:19roklenarcicso if someone puts a formatter instance there, then I will use it#2019-11-2212:20ikitommistring or a (predefined) keyword?#2019-11-2212:20ikitommi.. or a class, which is used.#2019-11-2212:20roklenarcicbut then they lose serializability, but that’s on them#2019-11-2212:20roklenarcicok… this was very helpful#2019-11-2212:21ikitommione thing there could be is to have a startup-time altering of a registry. via JVM options for example. It would be explicit, but still global.#2019-11-2212:22ikitommiplan is to pull schema defn and fn syntax helpers, can’t pass easily any custom options/registry there.#2019-11-2212:23ikitommi
(m/defn plus :- int? 
  [x :- int?, y :- int?] 
  (+ x y))
#2019-11-2212:25ikitommito support something like [:json-schema {:type "string"}] schema would require some way to change the one registry. could just be the var reference (`[json/json-schema {:type "string"}]`), but could be a JVM property to bootstrap some extra schemas into the registry… not sure if thi is a good idea#2019-11-2212:30ikitommi… or just malli.evil ns, which has all the public functions of other namespaces, but with the default-registry as an atom.
(require '[malli.evil :as evil])

(evil/register! ::int int?)

(evil/validate ::int 1)
; => true
#2019-11-2212:31ikitommi… or a JVM option to define the default-registry implementation. by default, it points to the immutable map, but could be swapped to atom impl. Bad Idea 🙂#2019-11-2414:21ikitommiabout closed/open schemas, comments welcome on this: https://github.com/metosin/malli/issues/31#issuecomment-557892061#2019-11-2417:33rschmuklerHey @ikitommi thanks for your response on https://github.com/metosin/malli/pull/119#2019-11-2417:34rschmuklerThis feature is currently blocking me, and I'm relatively familiar with malli's code at this point (lots of tracing to figure out how to implement the above - if you had ideas on what you were thinking in terms of implementation for #120 I could potentially submit a PR#2019-11-2417:34rschmuklerOne possible solution that comes to mind is to just extend the Schema protocol to have a -parent method and then invoke things as appropriate#2019-11-2417:35rschmuklerAlso, regarding the public m/parent function, perhaps we could also specify passing in a name to traverse up parents until one with the provided name is found#2019-11-2417:35rschmuklereg (m/parent schema :map)#2019-11-2417:44ikitommi@rschmukler I think the -parent is the way to go, but it might need some bigger refactoring on internals: when a Schema is created, the -father is not yet available, as we are creating the childs from the mothers “constructor”, so it needs to be a Delay or similar value. Derefon that while creating a child would either return nil or block indefinitely, depending on the implementation. So, would have to see the code after the change to know if that’s a good value for adding the (potential) complexity in.#2019-11-2417:44ikitommiif that is blocking you, you could add a custom decoder on the :map schema already?#2019-11-2417:45rschmukler@ikitommi I initially went down that path but then I had to do the key tracking to invoke the child schemas appropriately#2019-11-2417:45rschmuklerI was actually wondering if we might be better of implementing the coercions as a postwalk instead of a prewalk#2019-11-2417:46ikitommi
(m/decode 
  [:map {:decode/my-thing (fn [schema opts] ...do...thing...)} [:a int?] [:b int?]]
  {"a_kikka" 12, "b_kikka" 32}
  (mt/transformer {:name :my-thing))
#2019-11-2417:46rschmuklerThe issue becomes that if the map renames a key, the child schemas don't get mapped#2019-11-2417:47rschmukler(because the map now has a new key at a different path, and the transformers for the values are resolved via key, after the key is renamed#2019-11-2417:48rschmukler(Hence why I potentially suggest doing a postwalk instead of a prewalk for coercions) - is there any reason that it's currently implemented as a prewalk?#2019-11-2417:52rschmukler@ikitommi I have another PR that I'm going to submit that might be a simpler solution for what I'm trying to achieve - feel free to reject it#2019-11-2418:22ikitommiIt has to be prewalk, good example:
(m/decode
  [:multi {:dispatch :type
           :decode/string '(constantly #(update % :type keyword))}
   [:sized [:map [:type [:= :sized] [:size int?]]]
   [:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
  {:type "human"
   :name "Tiina"
   :age "98"
   :address {:country "finland"
             :street "this is an extra key"}}
  (mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
;{:type :human
; :name "Tiina"
; :address {:country :finland}}
#2019-11-2418:23ikitommithere will be two-way transformers using interceptos, one could put all the stuff into :leave to get postwalk#2019-11-2418:24rschmukler@ikitommi Interceptors is a great idea#2019-11-2418:35rschmukler@ikitommi are you open to applying the map transformer after the key and value transformer for the :map schema?#2019-11-2418:36rschmuklerie. Ideally I'd like to be able to have:
(is (= {:x_key "true", :y 1} (m/decode schema {:x true :y "1"}
                                             (transform/transformer
                                              {:name :custom
                                               :decoders
                                               {:map
                                                (constantly (fn [map]
                                                              (if-not (contains? map :x)
                                                                map
                                                                (-> map
                                                                    (assoc :x_key (:x map))
                                                                    (dissoc :x)))))
                                                'boolean? (constantly str)}}))))
#2019-11-2418:43rschmuklerActually this feels inconsistent - perhaps just implementing the interceptors is the way to go#2019-11-2420:13ikitommi@rschmukler what is your schema in the example?#2019-11-2420:15rschmukler[:map [:x boolean? :y int?]]#2019-11-2420:15rschmukler(although lets assume that I composed with transformer/json-transformer so that the number decodes correctly#2019-11-2420:16rschmuklerI realize that I can always pop an escape hatch and write a :custom/decode function but since I want to apply it on every map, it seems like a transformer is the idiomatic way to go#2019-11-2420:17rschmuklerI think interceptors solve everything so I'm trying my hand at implementing them right now#2019-11-2420:17ikitommibut, isn’t that m/encode you are doing?#2019-11-2420:18ikitommidecode should end up in valid values against the schema (eg. JSON->Schema), encode is Schema->JSON.#2019-11-2420:19rschmuklerThat's a buggy example haha#2019-11-2420:19rschmukler(mine above)#2019-11-2420:19rschmuklerIt is indeed an encode#2019-11-2420:20rschmuklerBut, none the less, I believe after the key is renamed during the encode, the subsequent value transformers will fail because they look up by the original key name#2019-11-2420:21rschmukler(ie. (if-let [entry (find m k)] (assoc m k (t (val entry))) I believe is the code in the map schema)#2019-11-2420:22ikitommitrue that. interceptors are needed#2019-11-2420:22rschmuklerYeah, hoping to have a PR today#2019-11-2420:22rschmukler😄#2019-11-2420:23ikitommiof interceptors? that’s a big one.#2019-11-2420:23rschmuklerRight now I'm having it so that -transformer and -value-transformer return {:enter _ :exit _} maps#2019-11-2420:24rschmuklerand then I'm updating the schema impls to use those on children as needed#2019-11-2420:24ikitommithere is https://github.com/metosin/malli/issues/114#2019-11-2420:24rschmukler
(defn- ->interceptor
  "Utility function to convert a transformer into an interceptor. Works with transformers
  that are already interceptors, as well as sequences of transformers"
  [transformer]
  (cond
   (fn? transformer) {:enter transformer :exit nil}
   (and (map? transformer)
        (or (contains? transformer :enter)
            (contains? transformer :exit))) transformer

   (coll? transformer) (reduce
                        (fn [{:keys [enter exit]} {new-enter :enter new-exit :exit}]
                          (let [enter (if (and enter new-enter)
                                        (comp enter new-enter)
                                        (or enter new-enter))
                                exit (if (and exit new-exit)
                                       (comp exit new-exit)
                                       (or exit new-exit))]
                            {:enter enter :exit exit}))
                        (map ->interceptor transformer))
   (nil? transformer) nil
   :else (throw (ex-info "Invalid transformer. Must be a function, sequence, or interceptor map"
                         {:value transformer}))))
#2019-11-2420:25rschmuklerI think that solves it, converting all transformers into an interceptor, even allowing for the vectors that you included#2019-11-2420:25rschmukler(in 114)#2019-11-2420:25rschmuklerthat map call should be a keep#2019-11-2420:26rschmuklerThen, like I said, basically all Schema impls need to be aware of :enter and :exit when returning their own interceptor (with :enter and :exit) themselves#2019-11-2420:26ikitommiawesome, looking forward to the PR. (`:exit`=> :leave would be coherent with interceptor libs)#2019-11-2420:26rschmuklerand then, I think encoder / decoder / encode / decode just need to call (comp exit enter)#2019-11-2420:26rschmuklerAh, yes, will rename the key, thank you!#2019-11-2420:26ikitommi👍#2019-11-2420:26rschmuklerDo you want me to rename the -transformer and -value-transformer fns to -interceptor and -value-interceptor?#2019-11-2420:27rschmukler(no opinions here)#2019-11-2514:15roklenarcicso what happens when you use mt/transformer to combine two existing transformers that have an encoder/decoder for same schema name?#2019-11-2514:17roklenarcicseems to me that the last encoder/decoder function wins#2019-11-2514:18roklenarcicso if you have a modifying transformer like strip-extra-keys-transformer and someone else defines a transformer for :map then one of them doesn’t work, if I am not mistaken#2019-11-2514:18ikitommishould put into chain (after interceptors)#2019-11-2514:19ikitommiI think#2019-11-2514:20roklenarcicchain?#2019-11-2514:20roklenarcicis this some new branch?#2019-11-2514:22roklenarcicshould I try to tamper with -transformer function to achieve a modifying effect#2019-11-2516:01rschmukler@roklenarcic https://github.com/metosin/malli/pull/122#2019-11-2516:02rschmuklerIt'd be reasonably easy to move the ->interceptor function in to have it compose if there's already existing transformers#2019-11-2516:03rschmukler@ikitommi it might be worth having it compose with both :enter and :leave steps#2019-11-2516:03roklenarcicyou have a comment block at the top there#2019-11-2516:03rschmuklerI just noticed that! Thanks!#2019-11-2516:05rschmuklerhttps://github.com/metosin/malli/pull/122/files#diff-f4026c83e07e6f542344d5945d310607R12#2019-11-2516:06rschmuklerSo basically on the transformer call where it calls ->interceptor - you could move that reduce internally and then later transformers in & options could just compose#2019-11-2516:08roklenarcicmaybe schema seems to be reporting wrong name in error message#2019-11-2516:08roklenarcichttps://github.com/metosin/malli/blob/master/src/malli/core.cljc#L556#2019-11-2516:17ikitommi.. about composing the transformers. What should happen when there are multiple transformers for same key? 1. last one wins (current) 2. all transformers with different name get chained 3. all get chained#2019-11-2516:18ikitommi
(mt/transformer mt/json-transformer mt/json-transformer)
1. normal json 2. normal json 3. json + json
#2019-11-2516:21ikitommi
(mt/transformer {:decoders {:map keywordize-keys}} mt/strip-extra-keys-transformer)
1. strip-keys wins 2. both 3. both
#2019-11-2516:22rschmukler@ikitommi I wouldn't do unique based off of name - I think for people unfamiliar with the implementation, you might not think that name matters much. Someone might write their own :json that they also compose with the built in json#2019-11-2516:22rschmuklerWe could do all unique functions though#2019-11-2516:22ikitommi
(mt/transformer {:decoders {:map keywordize-keys}} {:decoders {:map upper-case-keys})
1. upper-case wins 2. upper-case wins 3. both
#2019-11-2516:23rschmuklerI think both would be most useful#2019-11-2516:23rschmuklerthen you can write things like (mt/transformer keywords-as-strings capitalize-keywords)#2019-11-2516:24rschmukler(I could even see utility libraries existing with tons of tiny compositions of common transformer needs)#2019-11-2516:25rschmuklerAlso, I actually take back my comment on using unique functions#2019-11-2516:25rschmuklerI'd say let the user compose things however they want#2019-11-2516:36rschmukler@ikitommi if you'd like I can add that functionality to my interceptor PR#2019-11-2517:04eskosI almost commented on GitHub but not sure where this thought would actually fit, I started pondering that what if the data given to interceptor would be a zipper like navigation cursor instead? This would actually allow the interceptor to walk up/down/left/right the data structure when needed, without "bind to parent to get access" kind of tricks.#2019-11-2517:37rschmuklerPersonally I prefer the interceptor pattern as a balance of simplicity, power, and because it's common in other libraries in the ecosystem. The cursor is definitely much much much more powerful, but building it will be non-trivial. Specifically, the way schema expansion currently happens is sort of a pre-walk - so a child doesn't even have reference to its parent because when the child is instantiated the parent doesn't even exist yet.#2019-11-2517:42rschmuklerThe other thing of note is that right now all of the traversal / introspection of the schema happens at compile time (or, at encoder / decoder ) time. I'm not familiar enough with zippers to think through how mapping the functions produced by the zipper could then be composed into a transformer#2019-11-2517:55eskosI'm almost certain it would be runtime only and probably not that incredibly fast :)#2019-11-2518:00eskosBut I disagree on that sentiment of simplicity, as I don't see it as simple for any other than the simple use case - this parent awareness being a very good example of a sudden trickery situation, esp. with multi schemas where the parent doesn't necessarily dictate which child there is. It is always possible everyone's gotten things wrong so far... 🙂#2019-11-2518:01rschmuklerIt absolutely is!#2019-11-2518:01rschmuklerI agree with you 100%#2019-11-2518:02rschmuklerBecause the schema's are data, we at least have the escape hatch of compiling a different schema#2019-11-2518:03rschmuklerBut that feels "dirty" to me - not sure why... maybe bad habits from spec#2019-11-2518:04eskosI've never used spec beyond tutorials tbh, I disliked its global registry and nested specs too much.#2019-11-2518:04eskosSo maybe I'm not tainted/enlightened in that sense? 🙂#2019-11-2518:04rschmuklerLucky you 😉#2019-11-2518:05eskosBut anyway, it definitely is a concern that it would probably be really slow (in relation to rest of malli)#2019-11-2518:05rschmuklerThe tooling story for spec is pretty cool. Definitely something I intend to port over to malli once things stabilize a bit#2019-11-2518:05rschmukler
(>defn- wrap-vec
  "Wraps the given target into a vector. If its already in a sequence, it will
  be converted to a vector. If its a vector it will return itself. If its a value
  it will be inserted into a vector"
  [item]
  [any? => (s/coll-of any? :kind vector?)]
  (cond
   (vector? item) item
   (seq? item)    (into [] item)
   :else          [item]))
#2019-11-2518:05rschmuklerThat kind of inline annotation + optional checks is pretty useful#2019-11-2518:06rschmukleroptional instrumentation I should say#2019-11-2520:48ikitommiI would prefer the plumatic schema fn syntax:
(m/defn plus :- int? 
  [x :- int?, y :- int?] 
  (+ x y))
#2019-11-2520:49ikitommieffectively about the same.#2019-11-2522:17rschmukler@ikitommi interesting, why the preference? You like the type annotations right next to the variable name?#2019-11-2522:32rschmuklerI can likely make both valid. I'll probably just have different namespaces that you can import from#2019-11-2615:57ikitommi@rschmukler been using that with schema for 5+ years, no complaints. Cursive does static analysis for that + you can omit the type hints if you don't know/want to define those, e.g
(m/defn plus
  [x :- int?, y] 
  (+ x y))
#2019-11-2615:58ikitommiwill check the interceptor PR as soon as have extra time.#2019-11-2615:59ikitommithe chaining of encoders & decoders with mt/transformer, I think it's a good idea.#2019-11-2616:13ikitommihttps://github.com/danielcompton/defn-spec#2019-11-2617:02kszabothere is also: https://github.com/fulcrologic/guardrails/
#2019-11-2617:06ikitommiand https://github.com/jeaye/orchestra/blob/master/README.md#defn-spec#2019-11-2617:07ikitommiIs there something missing from the original plumatic schema syntax?#2019-11-2619:26rschmukler@ikitommi I’ve got a proof of concept. I’ve separated the parsing from the code generation (the generation just takes a map). This will allow different styles to work.#2019-11-2619:28rschmuklerRegarding the composition of interceptors, if you don’t mind, I may do it as a separate PR to keep thins more focused.#2019-11-2705:39eskosMy only gripe with Schema’s defn macros is that there’s no private versions and IIRC they sort of b0rked if you try to include type information or other metadata…but those are all fixable issues 🙂 And I might be misremembering things, I’m quite good at that.#2019-11-2807:40eskos@ikitommi As morning chore I made a Makefile for Malli 😛 mainly meant to replace the bin/kaocha and to make sure deps are installed. Since no one even asked for this, should I push this out or just keep to myself?#2019-11-2906:02eskos@ikitommi So I made a PR of this anyway, feel free to reject it if you disagree with my reasoning :)#2019-11-2906:10ikitommiThere are smarter on Metosin like @U1NDXLDUG doing our OS tooling stuff, looking forward to comments too.#2019-11-2906:31miikkabin/kaocha should not be replaced, it’s essential part of the Kaocha interface#2019-11-2906:33eskosPlease elaborate 🙂#2019-11-2906:33eskosI’m replacing the literal bash file, not kaocha itself.#2019-11-2906:34miikkaYes, but Kaocha is supposed to be used from command-line#2019-11-2906:34miikkaYou can do stuff like bin/kaocha --watch and bin/kaocha --focus test-symbol and stuff like that#2019-11-2906:34miikkaThat said, I wouldn't be against something like make test which would install the npm deps and then run the tests with bin/kaocha#2019-11-2906:35eskosAh, I see.#2019-11-2906:35eskosWell, that’s not a big adjustment 🙂#2019-11-2906:43eskosThere. I did the unfashionable and hid my crimes 😉#2019-11-2907:58miikkaI'm honestly not feeling very positive about your PR, but I'll try to review it at some point.#2019-11-2908:55eskosThat’s totally fine 🙂 Just wanted to push it out to clarify your stance on it.#2019-11-2819:59ikitommi@rschmukler Merged the interceptor-branch, thanks! Wanted to test myself if/how that works and re-formatted the code: https://github.com/metosin/malli/pull/124/files?w=1#2019-11-2820:01ikitommi
(is (= [24 48 8 10]
       (m/decode
         [:tuple
          {:decode/string (constantly {:enter (partial mapv inc), :leave (partial mapv (partial * 2))})}
          [int? {:decode/string (constantly {:enter (partial + 2), :leave (partial * 3)})}]
          [int? {:decode/string (constantly {:enter (partial + 3), :leave (partial * 4)})}]]
         [1 2 3 4] mt/string-transformer)))
#2019-11-2820:13ikitommidid an issue about the defn syntax, labelled “for discussion”.#2019-11-2820:56rschmukler@ikitommi Woo! Looks great 🍻#2019-11-2820:57rschmuklerI was just recently thinking that there are probably some interesting behaviors we should add for multi schemas. Eg. should we have a :dispatch/leave option? and then assume :dispatch and :dispatch/enter to be the same?#2019-11-2905:51ikitommihmm… I think the :dispatch doesn’t need a separate leave?#2019-11-2905:52ikitommias the dispatch is used to select the schema when going in, when returning, we don’t need the selection, we just go back the same route.#2019-11-2906:03ikitommiHmm,.. after mt/transformer chains the interceptors for the same schema key (instead of overriding), what if the names are also chained so that one could define multiple schema-based decode/encode via schema properties?#2019-11-2906:07ikitommie.g. (mt/transformer {:name :enter} mt/string-transformer {:name :leave}) would make a name chain of [:enter :string :leave] which would allow fine grained control of overriding things in the schemas:
(m/decode
  [:and {:decode/enter (constantly inc), 
         :decode/string (constantly (partial * 2))
         :decode/leave (constantly (partial + 10))} int?]
  1
  (mt/transformer {:name :enter} mt/string-transformer {:name :leave}))
; => 30
#2019-11-2906:08ikitommimore common case would be to “after normal decoding, do this too”:
(m/decode
  [:and {:decode/leave (constantly (partial + 10))} int?]
  "1"
  (mt/transformer {:name :enter} mt/string-transformer {:name :leave}))
; => 11
#2019-11-2906:08ikitommigetting complex, not sure if this is a good idea.#2019-11-2906:12eskosNot going to lie, I had to read that about 5 times to grok it…#2019-11-2906:13ikitommi
(m/decode
  [:and [int? {:decode/string '(constantly {:enter (partial + 2), :leave (partial * 3)})}]]
  "1"
  mt/string-transformer)
#2019-11-2906:13ikitommicurrently, that doesn’t work, as there is no way to compose the normal transformation and a custom one. if you define anything at schema-level, it overrides the default. might be ok and just needs to be documented.#2019-11-2906:21ikitommibtw, do you @borkdude have a defn-parser in some of your utility libs? would be great not need to reinvent the wheel with the schema defn -syntax, expecially if malli already depends on that functionality (sci or edamame)#2019-11-2906:25eskosYou can rip it out of Clojure quite easily (https://github.com/clojure/clojure/blob/clojure-1.9.0/src/clj/clojure/core.clj#L283)#2019-11-2906:31eskosAnd if that feels dangerous, I’ve made one incomplete implementation of defn parser with clojure.tools.reader a while back because I needed to analyze defn s (don’t ask) and ended up with this sort of bleh thing which eats defn s and produces data maps.#2019-11-2906:31eskosI’m sharing that just for the idea, it’s obviously not complete 😛 In fact I should continue this project, I’ve been sitting on it for far too long…#2019-11-2907:55borkdude@ikitommi there is a defn parser in sci of course but I haven’t thought about sharing it so it might be a little specific for sci #2019-11-2907:57borkdudeIn edamame there is a fn literal -> fn parser feature #2019-12-0218:21ikitommiintegrating into reitit (wip): https://github.com/metosin/reitit/pull/341#2019-12-0218:25ikitommiWe can create all the needed functions per schema & transformer ahead of time:
(->Coercer (m/decoder schema t)
           (m/encoder schema t)
           (m/validator schema)
           (m/explainer schema))
#2019-12-0516:10rschmuklerSorry about that @ikitommi!!#2019-12-0517:38roklenarcicSequential schema doesn’t seem to be working correctly:
(m/encode [:sequential {:encode/string (constantly #(clojure.string/join "," %))}
           string?]
          ["A" "B" "C"]
          mt/string-transformer)
          
=> (\A \, \B \, \C)
instead of expected: "A,B,C"
#2019-12-0517:39roklenarcicthe encode function correctly receives the input vector, and returns a string#2019-12-0517:39roklenarcicthe result should be a string#2019-12-0522:15rschmukler@roklenarcic I actually just ran into the same bug in a different manifestation! @ikitommi PR incoming...#2019-12-0523:05rschmukler@roklenarcic https://github.com/metosin/malli/pull/135#2019-12-0523:27roklenarcicWhat is the use case for the “interceptors” in Malli? What use cases did it solve? Because all I see is things getting more complicated and yet it adds no features. The interceptor change smells like cargo cult programming to me. Because http request/response frameworks have switched to interceptors, people think they are applicable to every problem. I just don’t see what it adds to Malli#2019-12-0523:29rschmuklerYou're right that in their current form they aren't that useful. The big draw to them would be if the :leave could execute in a more post-walk fashion. Right now, for simple use cases, they give an opportunity for a transformer to finalize a value after most things are done (ie. all :leave execute after all :enter have finished). I'm hoping that we can make it a true postwalk, in which case they become much more useful#2019-12-0609:07roklenarcicBut I was able to do that in the old system as well. My transform function was simply: call the child transformer and then finalize and return a value.#2019-12-0609:39ikitommi@roklenarcic could you show example how do you call the child transformer from a transformer?#2019-12-0609:40ikitommithe lib is at pre-alpha , so we can still rollback anything if there is unneeded complexity around#2019-12-0609:40roklenarcicAssuming you have in -into-schema schema' (-> children first (m/schema opts))#2019-12-0609:42roklenarcic
(-transformer [this transformer context]
            (let [child-xf (m/-value-transformer transformer schema' context)]
              (if child-xf
                (if (= :decode context)
                  (comp child-xf xf)
                  (comp xf child-xf))
                xf)))
#2019-12-0609:42ikitommiexample: you want to transform map after the keys have been renamed, using transformed keys. This is a good place to do it in leave.#2019-12-0609:44roklenarcicyou can create a transformer like this:
my enter logic
call child
my leave logic
#2019-12-0609:45roklenarcicthe difference is that in the interceptor pattern, calling the child (i.e. next in chain) is handled by the function that runs the whole thing#2019-12-0609:46roklenarcicbut in http request/response interceptors that code is doable because it is obvious how to handle calling the next interceptor in chain#2019-12-0609:46roklenarcicthere is 0 or 1 next interceptors#2019-12-0609:47roklenarcicin malli, you have maps and vectors and custom types, which means you cannot have standard function which will invoke next interceptors and combine results#2019-12-0609:53ikitommithe current interceptor runner is quite simple, but my idea was to use next version of sieppari, on which, the chain is a vector of IntoInterceptor instances, e.g. functions, maps or Interceptor Instances. The runner composes the (optimized) chain. This allows things like visual chain debugging, good perf and overall, and reuse.#2019-12-0609:54roklenarcicAre you talking about multiple interceptors on same schema?#2019-12-0609:54ikitommimanual chaining is not that handy if you want to inline the transformation via schema properties#2019-12-0609:54ikitommiyes#2019-12-0609:55roklenarcicor are you talking about composing say an interceptor on :sequential schema with those on child schemas#2019-12-0609:55ikitommihttps://github.com/metosin/malli/blob/master/test/malli/core_test.cljc#2019-12-0609:56ikitommiadded some tests to document how the chaining works now. Grep for :enter#2019-12-0609:57roklenarcicThose are serializable?#2019-12-0609:57ikitommiYes#2019-12-0609:57roklenarcic#() is serializable?#2019-12-0609:57roklenarcicI never knew#2019-12-0609:58ikitommiThanks to sci by @borkdude. Just quote them and it works#2019-12-0609:58roklenarcicas said, the biggest difference is that in request/response interceptor chains, they are strictly linear#2019-12-0609:59ikitommiMalli uses a safe/always-terminating subset of Clojure core with sci#2019-12-0609:59ikitommiis non-linear also needed?#2019-12-0610:01ikitommi#2019-12-0610:02ikitommithat's sieppari-based data visualization that we get for free with intercwptors#2019-12-0610:02roklenarcic:map-of and :vector combine enter and exit function in non-linear maner#2019-12-0610:03ikitommihmm. haven't had time to check out that. Could you please explain?#2019-12-0610:04roklenarcicyes, when writing -transformer function you would expect that with interceptors one doesn’t have to do the composing with the next interceptor in chain#2019-12-0610:05roklenarcicbut you still have to manually write -transformer function to compose your interceptor with the others#2019-12-0610:06roklenarcice.g. in :vector you need to write code that will return a map {:enter … :leave ...} such that it correctly composes its enter/leave logic with the elements enter/leave logicks#2019-12-0610:10roklenarcicI guess I’ll see where this is going#2019-12-0610:22ikitommiI see. Need to think about this when at computer.#2019-12-0610:24roklenarcic@rschmukler yesterday I hit this snag https://clojurians.slack.com/archives/CLDK6MFMK/p1575567535304400#2019-12-0610:25roklenarcicyou made a patch#2019-12-0610:25roklenarcicI found another problem#2019-12-0610:26ikitommiIt was @rschmukler's patch#2019-12-0610:26roklenarcic
(m/encode [:sequential {:encode/string (constantly {:leave #(clojure.string/join "," %)})}
           string?]
          ["A" "B" "C"]
          mt/string-transformer)
Error printing return value (NullPointerException) at clojure.core/map$fn (core.clj:2755).
null
#2019-12-0610:27roklenarcicso if I wrap it in :leave I get NPE#2019-12-0610:27roklenarcicI know it was his patch that’s why I @ him#2019-12-0610:31ikitommiplease report all issues, clearly need to fix or rethink this. If you have suggestions how to make the transforming good with/without interceptors, while supporting the schema property based extension (serializable), I'm all ears.#2019-12-0610:47roklenarcicmy question before thinking about this is what is the desired logic for this part: • if multiple transformers define encoder/decoder for same symbol/schema what is desired, last one wins? • if schema use-site property map defined encode decode keys, what is the desired interaction with existing encode decode operation already defined by schema type itself, full override? does it also override the construction in -transformer? #2019-12-0615:49rschmukler@roklenarcic Try the latest of that patch. Fixed that issue.#2019-12-0615:49rschmuklerThat was actually a lingering issue from a work-around that I did because fwrap wasn't skipping on non-collection input. Should be good now#2019-12-0620:05ikitommimy 2 cents: • transformer should not have only one -transformer-name, but instead, a chain of names. strip-extra-keys should have an unique name • with mt/transformer, one can create a chain of transformations, wehre both names and encoders & decoders are chained together and the chain of names is used for schema property based lookups. Any schema-defined will override the one defined in the transformer • encoders and decoders are defined as a transformation function of type schema opts => IntoInterceptor. This allows any transformation function to access the schema in question at “compile time”, allowing fast “drop extran keys” etc. • interceptor is to describe a transformation step. There is a IntoInterceptor Protocol, which is extended for a function (maps to :enter), a map, Interceptor Record or a chain of IntoInterceptor . Malli could use sieppari later, as it already defines all of these.#2019-12-0620:06ikitommiIn most cases, users don’t have to worry anything about the interceptor machineny, just use plain functions and it works. When you need more batteries on what happens when, you can either a) start using interceptors (enter & leave) b) add more steps into the transformation chain c) both#2019-12-0620:07ikitommiSimplest thing:
(m/decode
  [string? {:decode/string '(constantly #(str "olipa_" %))}]
  "kerran" mt/string-transformer)
; => olipa_kerran
#2019-12-0620:07ikitommiWith Interceptor:
(m/decode
  [string? {:decode/string '(constantly {:enter #(str "olipa_" %)
                                         :leave #(str % "_avaruus")})}]
  "kerran" mt/string-transformer)
; => "olipa_kerran_avaruus"
#2019-12-0620:08ikitommiA custom transformaton chain:
(m/decode
  [:and
   {:decode/before '(constantly inc)
    :decode/after  '(constantly (partial * 2))}
   int?]
  1
  (mt/transformer {:name :before} {:name :after}))
; => 4
#2019-12-0620:10ikitommiboth:
(m/decode
  [:and
   {:decode/before '(constantly {:enter inc
                                 :leave (partial * 2)})
    :decode/after  '(constantly {:enter (partial * 4)
                                 :leave inc})}
   int?]
  1
  (mt/transformer {:name :before} {:name :after}))
; => 18
#2019-12-0620:11ikitommiusing a chain:
(m/decode
  [:and
   {:decode/before '(constantly [inc inc #(* % 2) inc])}
   int?]
  1
  (mt/transformer {:name :before} {:name :after}))
; => 7
#2019-12-0620:30ikitommiAs we’ll soon lose the Slack History, wrote it down as issue: https://github.com/metosin/malli/issues/136#2019-12-0620:32ikitommialso, idea to remote the schema opts => interceptor intermediate step in favor of reitit-style interceptors that have a separate compile-step. Sounds even more complex, but actually would simplify things.#2019-12-0620:32ikitommiwith it, the simple case:
(m/decode
  [:and
   {:decode/after 'inc}
   int?]
  "1"
  (mt/transformer {:name :before} mt/string-transformer {:name :after}))
; => 2
#2019-12-0620:33ikitommithe complex case:
;; mounts only if there :multiplier property in schema
(def multiply-interceptor
  {:name ::multiply
   :compile (fn [schema _]
              (if-let [multiplier (:multiplier (m/properties schema))]
                (partial * multiplier)))})

;; the interceptor does not mount as the `:multiplier` is missing:
(m/decode
  [:and
   {:decode/after multiply-interceptor}
   int?]
  "2"
  (mt/transformer {:name :before} mt/string-transformer {:name :after}))
; => 2

(m/decode
  [:and
   {:decode/after multiply-interceptor
    :multiplier 10}
   int?]
  "2"
  (mt/transformer {:name :before} mt/string-transformer {:name :after}))
; => 20
#2019-12-0620:35ikitommi… and for the question: > does it also override the construction in -transformer no.#2019-12-0700:27roklenarcichm ok… but should the`-transformer` function return a function that returns transform function? Because so far it just seems that it brings a lot of constantly use#2019-12-0700:29roklenarcicI see so much constantly use that I wonder what is the other case (i.e. non-constantly)#2019-12-0703:58ikitommi-transformer already sees the schema, so it can return directly an .... interceptor? The later comment shows how we can get rid of vobstantly in the schema properties too: https://github.com/metosin/malli/issues/136#issuecomment-562731476#2019-12-0711:15ikitommithe current internal vocabulary is quite messy, would these be good names for the future: * method :decode :encode * stage :enter :leave * context :json :string :before :after ...#2019-12-0712:05ikitommi:enter/:leave and chain of contexts enable same kind of things. Wondering do we need them both…
(def transformer (mt/transformer {:name :before} mt/string-transformer {:name :after}))

(m/decode 
  [:and 
   {:decode/after 'inc} 
   int?] 
  "1" 
  transformer)
; => 2

; :enter
;  :before
;  :strip-keys
;  :string => ->int "1" => 1
;  :after => inc 1 => 2
; :leave
;  :after
;  :string
;  :strip-keys
;  :before

(m/decode 
  [:and 
   {:decode/before {:leave 'inc}}
   int?]
  "1" 
  transformer)
; => 2

; :enter
;  :before
;  :strip-keys
;  :string => ->int "1" => 1
;  :after
; :leave
;  :after
;  :string
;  :strip-keys
;  :before => inc 1 => 2 
#2019-12-0716:51ikitommiwelcome @viesti simple_smile#2019-12-0716:52viesti👋#2019-12-0716:53ikitommiDid a small spike, seems to work, so PR out. Simplifies the Transformer Protocol and enables chaining of tranformers (instead of overriding schema-based encoder & decoders like before). Also, better naming of things https://github.com/metosin/malli/pull/137#2019-12-0716:53ikitommi
(m/decode
  [int? {:decode/before '(constantly {:leave inc})
         :decode/after '(constantly (partial * 2))}]
  "10"
  (mt/transformer
    {:name :before}
    mt/string-transformer
    {:decoders {'int? (constantly inc)}} ;; anonymous
    {:name :after}))
; => 23

;; :enter
;;  :string   "10" => 10
;;  anonymous   10 => 11
;;  :after      11 => 22
;; :leave
;;  :before     22 => 23
#2019-12-0917:11rschmukler@ikitommi just confirming that with the changes, a custom :string/decode on a schema entry will still fully overwrite the default mt/string-transformer for the type, yes? I think that escape hatch is quite useful.#2019-12-0917:12ikitommi@rschmukler yes, will override, no changes there#2019-12-0917:13rschmuklerWonderful! Reading through the code but it looks great so far#2019-12-0917:13ikitommijust chains all the different names.#2019-12-0917:14rschmuklerHmmmm - do we even need transformer-chain protocol method?#2019-12-0917:14rschmuklerIs it just for introspection?#2019-12-0917:14rschmuklerI guess we're using it to assemble the interceptor chain#2019-12-0917:14ikitommioh, good point.#2019-12-0917:15rschmuklerBut I think we could move all of that into the transformer function and then simplify Transformer to a single value-transformer method, which could be nice#2019-12-0917:18rschmukler@ikitommi do you have any thoughts on https://github.com/metosin/malli/pull/132#2019-12-0917:19rschmuklerIt ended up being critical for my application (the ability to encode the renaming of keys for future transformers) but I'm open to alternative approaches if you've got other ideas you want me to explore#2019-12-0917:19rschmuklerPart of me thinks using metadata is odd - part of me thinks that this is the perfect use case for it#2019-12-0917:20rschmuklerie. it is metadata about how the new value was created... so maybe it makes sense?#2019-12-0917:21rschmuklerOne thing we may have to do though, if we end up sticking with that API, is merge the metadata about transformers as we compose them#2019-12-0917:28rschmuklerAlso, random aside, but malli is Finish, correct?#2019-12-1006:30eskosIt’s Finnish for model (in both shape, example and fashion model sense), yes 🙂#2019-12-1006:32eskoshttps://www.sanakirja.org/search.php?q=malli&amp;l=17&amp;l2=3#2019-12-1007:13Toni Vanhalaand mälli is chewing tobacco 😉#2019-12-1010:08eskosor heavy impact 🙃#2019-12-1016:05roklenarcicis there some way to have a schema like `
s/keys*
#2019-12-1016:05roklenarcicNamely I’d like to generate sample function arguments, so I need a schema that describes function input#2019-12-1016:06roklenarcicbut the function is defined as [arg1 arg2 & {:keys [arg3 arg4]}#2019-12-1016:23ikitommi@roklenarcic there are no sequence schemas like that, yet. Feature Request welcome.#2019-12-1016:24ikitommidefinetely interested in having that, ideas on impl also welcome.#2019-12-1016:27ikitommi@rschmukler haven't had time to check the rename keys, will do that when next time doing anything for malli. Good thing is that you can depend your fork that's to deps.#2019-12-1017:54rschmukler@ikitommi checking your latest recommendation but I think I'll be able to break it 😛#2019-12-1017:56rschmuklerAh, I think I figured it out!#2019-12-1018:37rschmukler@ikitommi updated https://github.com/metosin/malli/pull/132 with something less abstract so you can see why it's difficult as currently implemented#2019-12-1018:42ikitommithanks, will try to find an elegant way to resolve.#2019-12-1106:03ikitommiChecked the code., the build-transformer doesn’t work correctly. I think the same problem is with all sequential schemas: the chain in :leave shoud be in reverse order, so first apply the (child) value-transformers, then keys and last the map. This way, when changing the keys in :leave , it’s the last thing that happens.#2019-12-1106:06ikitommiAlso, all the three type of transformers should be created outside of the build-transformer fn. Current impl creates all three two times (both in enter & leave) causing potential state-bugs with -value-transformerimpls as the enter & leave have different instances attached to them.#2019-12-1106:06ikitommiI’ll write an issue, PR welcome.#2019-12-1106:07ikitommiAfter that, this is the way to rename keys locally on encode:
(m/encode
  [:map
   [:foo [:map
          [:foo int?]
          [:bar int?]]]
   [:bar int?]]
  {:foo {:foo 2 :bar 1} :bar 3}
  (transformer
    {:encoders {'int? (constantly inc)
                :map (constantly {:leave #(set/rename-keys % {:foo :oof :bar :rab})})}}))
; => {:oof {:oof 3, :rab 2}, :rab 4}
#2019-12-1106:08ikitommiafter the upcoming interceptor :compile, will reduce to:
(m/encode
  [:map
   [:foo [:map
          [:foo int?]
          [:bar int?]]]
   [:bar int?]]
  {:foo {:foo 2 :bar 1} :bar 3}
  (transformer
    {:encoders {'int? inc
                :map {:leave #(set/rename-keys % {:foo :oof :bar :rab})}}}))
; => {:oof {:oof 3, :rab 2}, :rab 4}
#2019-12-1106:08ikitommiwhich will, I hope, solve your case @rschmukler elegantly enough?#2019-12-1115:25ikitommiwould be interesting to see how malli would work on the dev-tooling space. Related to https://www.reddit.com/r/Clojure/comments/e95pr5/data_structure_shape_hints_and_intellisense/#2019-12-1115:54rschmukler@ikitommi nice! Good catch on build-transformer being called twice and its impact on local state! Changing the order of the chains is also perfect - that'll make it much more like a postwalk on leave#2019-12-1118:04rschmukler@ikitommi I've got the first prototype of my defn-spec / ghostwheel clone. For now I'm doing it as a separate project (called aave). Will be releasing a first version shortly#2019-12-1118:06rschmuklerThe syntax is abstractable though, so you can use schema style too if you want. Things that it currently supports are purity detection, configurable hooks for input and output validation at both compile time and run time. Next up is benchmarking.#2019-12-1118:06rschmuklerI also thought it'd be cool to be able to do something like#2019-12-1118:08rschmukler
(defn add-name-key
  [a]
  [:map => (+ a [:name string?])]
  (assoc a :name "Bob")
Where + symbol basically expands out to "merge schema from parameter a "
#2019-12-1118:08rschmuklerMight be overkill haha - but it could enable deep compile-time tracking of types#2019-12-1118:13ikitommi@rschmukler sounds awesome, and what a name! 👻#2019-12-1118:13rschmukler😄 I'm glad you like it#2019-12-1118:14bedersok, let's try this again. How would you write a function that returns a malli spec to test if a string is of length n?#2019-12-1118:15bedersI'm getting this far:#2019-12-1118:15beders
(defn fixed-length? [n]
  [:and 'string? [:fn `(~'fn [~'val] (~'= ~n (~'count ~'val)))]]
  )
#2019-12-1118:15bedersthat gives me:
(preds/fixed-length? 10)
=> [:and string? [:fn (fn [val] (= 10 (count val)))]]
#2019-12-1118:18rschmukler@beders Do you need it to all be quoted? Malli should take care of that for you. if not, you can just do:
(defn fixed-length? [n]
  [:and string? [:fn #(= n (count %)]])
#2019-12-1118:18bedersbut my code doesn't look very nice. Lots of quoting to prevent backtick from full name resolution#2019-12-1118:19bedersyour version captures n#2019-12-1118:19bedersI think#2019-12-1118:21beders
(defn fixed-length? [n]
  [:and string? [:fn #(= n (count %))]])
=> #'sql-to-malli.integration/fixed-length?
(edn/write-string (fixed-length? 10))
=>
"[:and string? [:fn #object[sql_to_malli.integration$fixed_length_QMARK_$fn__10416 0x576cee60 \"
#2019-12-1118:21rschmuklerAh I suppose that's true!#2019-12-1118:21bedersif I don't use the extension unquote-quoting, I get the fully resolved names#2019-12-1118:22rschmuklercan you do#2019-12-1118:22beders
(defn fixed-length? [n]
  [:and 'string? [:fn `(fn [val] (= ~n (~'count val)))]]
  
  )
(fixed-length? 10)
=> [:and string? [:fn (clojure.core/fn [clojure.core/val] (clojure.core/= 10 (count clojure.core/val)))]]
#2019-12-1118:22bederswhich kinda works, just not very readable#2019-12-1118:23bederstrying to stick with the data-only mantra here#2019-12-1118:23bedersso no capturing#2019-12-1118:23rschmuklerBut you need your schema to be serializable?#2019-12-1118:24bedersI'm working on a library that takes a SQL database schema and spits out malli#2019-12-1118:24rschmuklerI'd consider adding it to your registry using fn-schema#2019-12-1118:24bedersI can probably do without serialization#2019-12-1118:24ikitommiyou can single-quote the fn#2019-12-1118:24bedersif I single-quote the fn, I'm capturing n#2019-12-1118:24rschmukler@ikitommi he needs to get the 10 in there though#2019-12-1118:25rschmuklerbut with a custom registry, you could just do [:and 'string? [:fixed-length 10]]#2019-12-1118:25bedersyes, adding it to the registry might be the only sane choice for now#2019-12-1118:26bedersthat would make my library a runtime dependency#2019-12-1118:27bedersI was hoping I could run this on a database schema, copy the resulting malli schemas and then run with them#2019-12-1118:27bederslooks like I need to decide on various trade-offs#2019-12-1118:27ikitommia :fn variant that takes extra args from properties?#2019-12-1118:27bedersjust wanted to get some ideas on code generation#2019-12-1118:28bedersthat make for readable schemas#2019-12-1118:28bedersI like [:fixed-length 10] though. Very readable#2019-12-1118:30bedersthanks for the ideas!#2019-12-1118:30rschmuklerOh you know#2019-12-1118:31rschmukleryou might be able to do#2019-12-1118:32rschmuklerNope, never mind 😄#2019-12-1118:50ikitommiJSON Schema defines extra attributes for numbers, strings and arrays. Malli could have those too? [string? {:min 10, :max 10}]#2019-12-1118:51ikitommiAlso, malli schema properties should be defined with malli schemas to get nice docs & validation of those too.#2019-12-1118:54ikitommi@beders if the runtime dependwncy would be ok, you could also just use the Schema Var instead of registry: [my-lib.schemas/fixed-lenght 10]#2019-12-1118:58bedersGood point. I mean, technically, this works too:
(m/validate [:re #"^.{4}$"] "bubu")
#2019-12-1118:58bedersbut probably a bit overkill 😉#2019-12-1118:58ikitommithe fn can also be a string btw, so this should work (injection warning!!): [:and string? [:fn (str "#(= " n " (count %))")]]#2019-12-1119:00bedersah, didn't know that. Thx!#2019-12-1119:02ikitommilooking forward to seeing the lib.#2019-12-1201:42bedersit looks like m/fn-schema doesn't allow for any other parameters (or children)#2019-12-1201:43bedershere's what I tried:
(defn max-length?
  ([n s]
   (and (string? s) (< (count s) n)))
  )
(def default-registry (merge m/default-registry {:max-length (m/fn-schema :max-length preds/max-length?)}))
#2019-12-1201:44bedersI then should be able to say this, right?
(m/validate [:max-length 4] "bubu" {:registry default-registry})
#2019-12-1201:45bedersbut alas
#error{:cause ":malli.core/child-error",
       :data {:type :malli.core/child-error, :data {:name :max-length, :properties nil, :children (4), :min 0, :max 0}},
#2019-12-1201:45bederswhich - when checking the code - happens in fn-schema#2019-12-1201:46bedersit seems -partial-fn-schema allows for additional parameters, but is private#2019-12-1201:46bedersalso, fn-schema doesn't pass on properties#2019-12-1201:48bedersany hints on how to register custom predicates with params would be awesome#2019-12-1205:41JohanIs Malli a good place to handle default value ?#2019-12-1207:09ikitommi@johan178 sure. could be just :default property and a pre-defined transformer to use that for missing values. metosin/schema-tools has just about that.#2019-12-1207:10ikitommineed to add capability to transform all schemas, so need a small development before it can work. Please raise an issue.#2019-12-1219:47rschmuklerHey @ikitommi I'm in the process of fixing up transforms for independent leaves and enters (and reverse order). I have a few questions for you while I'm in here...#2019-12-1219:51rschmukler1. For the collection transformer, right now fwrap auto-coerces things into the appropriate type. This happens after the value-transformer is called currently - meaning that I don't think it's possible for the value-transformer to change it's collection type on encode because it'll always be coerced back. Should we keep this behavior? 2. the map-of transformer currently calls transform keys and transform values at the same time as it iterates through the map. I think it's fine to keep this for enter and leave? ie. it'll basically be enter: transform-map -> transform k 1 -> transform v 1 -> transform k 2 -> transform v 2 and leave: transform k 1 -> transform v 1 -> transform k 2 -> transform v 2 -> transform map 3. The collection transformer currently does a single step of building the collection and mapping the values. We probably want to do enter: coll-transform -> transform children and leave: transform-children -> transform#2019-12-1221:03ikitommi1) I think the fwrap should be removed and moved under Transformer decoders. e.g. as the arrays are always mapped to Clojure vectors in JSON decoding (by json parsers in both clj & cljs), with mt/json-transformer there should be no transformation needed for :vector, only for other sequence types. The currently silly mt/collection-transformercould have decoders for all collection types.#2019-12-1221:03ikitommi2 & 3 👍#2019-12-1221:04rschmuklerRe. 1 I want to land this PR w/o that then if you don't mind. It's already huge#2019-12-1221:04ikitommisure#2019-12-1221:05rschmuklerThe other thing that we don't do much, that I'm going to add, if you think it's a good idea, is ensuring that the type doesn't change between transforms#2019-12-1221:05rschmuklerie. if a map encodes itself to an integer, we still try and call reduce-kv on it for the key and value transformers#2019-12-1221:05rschmuklerWe likely shouldn't do that#2019-12-1221:06rschmuklerActually - I'm going to keep this focused at just the changes to fix #140 and the encode decode order - we can add more later#2019-12-1221:07ikitommibtw, the 3 might have a big effect on the perf?#2019-12-1221:08rschmuklerI'm still using transducers where possible#2019-12-1221:08ikitommidepending on how the collection is transformed, retaining the type automatically or not#2019-12-1221:08rschmuklerand using fempty etc#2019-12-1221:08ikitommicool. Sorry, Need to go.#2019-12-1221:08rschmuklerI tried to be mindful of the performance implications - but a close look is definitely a good idea!#2019-12-1221:09rschmuklerCool, PR incoming#2019-12-1221:09rschmuklerCatch you later!#2019-12-1304:53JohanCan a defined transformer access the schema ? So given [:map [:key {:option :cool} keyword?]] Can a transformer access the [:key {:option :cool} keyword?]] data ?#2019-12-1306:12ikitommi@johan178 yes, the current transformer is of type schema opts => value => value, so:
[:map {:decode/string (fn [schema opts] 
                        (let [cool (:option (m/properties schema))]
                          (fn [value] …)))}]
#2019-12-1306:13ikitommithe syntax will change before hitting first public to:
[:map {:decode/string {:compile (fn [schema opts] 
                                  (let [cool (:option (m/properties schema))]
                                    (fn [value] …)))}}]
#2019-12-1306:14ikitommihttps://github.com/metosin/malli/issues/136#issuecomment-562731476#2019-12-1316:35rschmukler@ikitommi Thanks for the quick PR review! Would you be open to me adding .nrepl-port and .dir-locals.el to gitignore? Alternatively if you want me to commit a .`dir-locals.el` it would make it so that new people contributing would automatically adhere to your style preferences#2019-12-1316:46ikitommi@rschmukler sure, go ahead, either way#2019-12-1316:46rschmukler:thumbsup:#2019-12-1317:29eskos.dir-locals.el is only relevant to cider users, so I wouldn't commit them. In general tooling/IDE config files shouldn't live in source repos 😛#2019-12-1317:48rschmukler.dir-locals.el is relevant to all emacs users#2019-12-1317:49rschmuklerBut I'm happy to not commit it - just could save PR time of people having to adjust their formatting to the project style#2019-12-1317:59rschmuklerhttps://github.com/teknql/aave - First rendition of Aave!#2019-12-1318:08ikitommiagree on Esko for now, so let’s ignore the files.#2019-12-1318:09ikitommiCongratz on Aave!! I’ll add it to README as soon as the PRs are merged (just did the interceptor :compile, will merge it after yours @rschmukler).#2019-12-1318:09rschmuklerWoo! Thanks! Love the momentum we've got going on!#2019-12-1318:22rschmukler@ikitommi What're your thoughts for potentially adding support for a dynamic variable for either full on malli options, or perhaps something like *registry-extensions* which is usually nil but is automatically merged into default-registry when set. I sometimes find myself having to thread malli options through things, or wrapping malli functions with my own namespace with a custom registry, etc. Since it's mostly for compile-time code anyway, the performance implications shouldn't matter much. Part of me feels like it might be too "magical" but the other part makes me feel like it could improve ergonomics#2019-12-1318:33ikitommineed to think about that.#2019-12-1318:34ikitommi@rschmukler do you have plan to fix the 142? (the is the x bug IMO + the formatting)#2019-12-1318:35rschmuklerI think I already did no?#2019-12-1318:35rschmuklerI force pushed updates an hour or two ago#2019-12-1318:36rschmuklerLet me know if it needs anything. The x bug wasn’t caught by tests because map-of tests were disabled so I also fixed those #2019-12-1318:37ikitommioh, the review panel just doesn’t show the updated. my bad. will merge.#2019-12-1318:38rschmuklerAwesome!#2019-12-1318:40ikitommicould you check https://github.com/metosin/malli/pull/146 ?#2019-12-1318:44rschmuklerYep, will take a look now#2019-12-1318:53rschmuklerLooks good! Sorry I missed some indents#2019-12-1319:09ikitomminp, off to weekend. cheers.#2019-12-1510:57ikitommiAbout to add :default-decoder and :default-encoder to Transformer options: used if nothing else matches. Makes things like “adding defaults” easy. Comments welcome on that: https://github.com/metosin/malli/issues/143#issuecomment-565798326#2019-12-1520:30ikitommiAnd here’s the PR. Rewrote the malli.core naming to be more corerent, more reuse in chaining & collection schemas now accept nil in transformations. https://github.com/metosin/malli/pull/148#2019-12-1617:25rschmukler@ikitommi awesome cleanup on malli.core. Code looks 😍#2019-12-1617:31rschmuklerJust rebased https://github.com/metosin/malli/pull/147#2019-12-1617:42ikitommiactually, the logic to convert from keywords should be somewhere. JSON Parsers keywordize the keys. If keys are defined to be int?, They are seen as :12 etc. The string->int doesn't work here#2019-12-1617:43rschmuklerAh! I can look into re-adding that to the json transformer then#2019-12-1617:43ikitommi.. and string too#2019-12-1617:44rschmuklerOkay, should I just add it to the string transformer then?#2019-12-1617:44rschmukler@ikitommi I think we should consider dropping the schema-based type checks and instead leave that to the transformer functions. ie. sometimes we check (if (or (map? x) (nil? x)) (transform x)) - especially on decode, it might not be the same shape at all...#2019-12-1617:44rschmuklerConsider a JSON body that returns [x y z] and wanting to decode that into a map... or similarly wanting to use malli to decode CSV rows into maps#2019-12-1617:45rschmukler
(decode [:map {:decode/row (fn [row]
                               {:name (nth row 0)
                                :age (nth row 1)})}
           [:name string?]
           [:age int?]]
          ["Bob" "42"]
          (malli.transform/transformer
           malli.transform/string-transformer
           {:name :row}))
#2019-12-1617:45rschmuklerIdeally that'd work#2019-12-1617:47rschmuklerI think it should be the transformer's responsibility to handle (or not handle) arbitrary data#2019-12-1618:04rschmukler@ikitommi re-added the coercion behavior to string + json transformer. Still marked the commit as breaking since it'll break people who were relying on the keyword->string coercion in their transformers... but I updated the message + commit body to tell as much.#2019-12-1618:44ikitommi@rschmukler agree on pushing the checks to transformers. Currently, the check is only done before the whole chain. If the first step changes the type, with the current solution, the next will blow anyway.#2019-12-1618:45ikitommi147 looks good. I was told not to pull in before the EPL2 license is in place, need still few approvals.#2019-12-1618:47ikitommiI think the current :malli.core/map-key target (and impl) can be removed - same can be done with anonymous transformer with :map target.#2019-12-1619:25rschmukler@ikitommi Alright sounds good. I think I've got a elegant solution to the transformer chain checks#2019-12-1621:02ikitommihttps://github.com/metosin/malli/pull/149#2019-12-1622:22rschmukler@ikitommi https://github.com/metosin/malli/pull/150 😄#2019-12-1716:26rschmuklerAny word on when the code freeze will end?#2019-12-1718:41miikkaAs soon as hear back from everyone. I've got 4/6 confirmations so far.#2019-12-1721:05rschmuklerWoo!#2019-12-1816:15rschmukler@ikitommi I haven't looked at the code at all yet, but how difficult do you think it would be to extend the malli.provider to be pluggable / use transformers? ie. Ideally I could specify a transformer and it'd try transformations in there, and it'd check types that exist in the registry when called#2019-12-1816:19rschmuklerLooking at the code - doesn't look like it'd be too hard to make it take a transformer!#2019-12-1816:20rschmuklerThe only thing would be handling the fact that multiple transformers could pass and then having to keep track of which one was most successful#2019-12-1816:20rschmuklerBut we are kinda already doing that with the type anyway#2019-12-1909:05miikkaMalli is now under EPL-2.0 and the code freeze has ended.#2019-12-1913:43pezI should have looked for a malli channel, of course!#2019-12-1915:32ikitommi#2019-12-1915:32ikitommi#2019-12-1915:33ikitommiwelcome @pez! looking forward to your validation lib 🙂#2019-12-1915:51pezAwesome. I am using 1 right now. It was 3 I was trying to do but didn't figure out about the :error/path property. Will try that now.#2019-12-1915:53pezDon't hold your breath about that lib. 😃 But who knows, we do have many kinds of form scenarios in our app so maybe I get reason to really figure this out.#2019-12-1917:42rschmuklerhttps://github.com/weavejester/integrant/pull/60 - fingers crossed 🙂#2019-12-2006:55ikitommi@rschmukler malli.provider, transformer, so it could if there is a use case for it. Some kind of extension mechanism would be good anyway, adding inferring to :tuple and :multi currently require a lot of internal refactoring. Not easy to add inferrers to own schemas.#2019-12-2006:57ikitommias it's not complete yet, would be good to add the missing inferrers and options to controls those and see what kind of architecture evolves out of those.#2019-12-2007:02ikitommimaybe options like {:infer {:multi {:stats ...,. :schema ..., :opts {:dispatch #{:type 'first}, :min 2, :propability 0.90}}},#2019-12-2007:04ikitommie.g. callback to participate in extracting stats out of samples, and converting those into schemas + opts for what kind of multis to look and how many samples needed#2019-12-2007:06ikitommimight be better to start with just options and hard-code the impl, and try to extract the callback mechanism later.#2019-12-2016:27rschmukler@ikitommi the other thing I was thinking about is the ability to code options. ie. imagine a LocalDateTime parser that has an property of :date/format "MM/dd/yyyy" - ideally it'd be good to specify "try these formats when you try to parse things as a date" etc
#2019-12-2016:27rschmuklerStill thinking about how it could look#2019-12-2018:29ikitommiidea: If the inferrer is good, there could be a tool that records all args and return values for functions and infers defn-schemas out of those. I think @ambrosebs did something similar with spec (using core.typed)#2019-12-2018:38pezLovely idea!#2019-12-2023:42borkdude@ikitommi that's awesome. this information can then also be used to generate clj-kondo type annotations#2019-12-2114:12ikitommi2 months to ClojureD, would be awesome to have some clj-kondo integration for that.#2019-12-2114:13ikitommiAlso, to finalize this…#2019-12-2114:17ikitommidon’t know if it’s problem in fipp or in my cursive setup, but some some broken ansi escape codes threre.#2019-12-2116:05borkdude@ikitommi This is the format the clj-kondo currently understands: https://github.com/borkdude/clj-kondo/blob/master/doc/types.md Any tool that can generate that format from whatever data available can integrate with clj-kondo#2019-12-2119:07ikitommiwell, that’s looks simple.#2019-12-2716:12ikitommidid a quick cleanup, without too much thinking, so comments welcome, https://github.com/metosin/malli/pull/152#2019-12-2809:38ikitommireitit coercion with malli 90% done#2019-12-2818:28rschmukler🔥🔥🔥#2020-12-3003:41JohanI'm trying to implement the ideas developed on this talk with Malli. Anyone else is splitting attributes schemas and selections ? source: https://www.youtube.com/watch?v=YR5WdGrpoug#2020-12-3008:54ikitommi@johan178 some discussion here: https://github.com/metosin/malli/issues/95#2020-12-3009:04ikitommiAlso: https://github.com/metosin/malli/issues/14#2020-12-3009:08ikitommithat said, it would be trivial to write a custom select schema, allowing serializable selects:
[:select 
 [:map [:x int?] [:y int?]]
 [:x]]
#2020-12-3110:57borkdudeDoes malli have something like s/conform for s/cat?#2020-12-3111:11ikitomminot yet, but it will come. Can't recall if there was an issue of conform yet, or just discussed#2020-12-3112:44ikitommi12 added lines later:
(explain
  [:map {:closed true}
   [:xy
    [:map {:closed true}
     [:x int?]
     [:y int?]]]]
  {:xy {:x 1, :y 2, :EVIL "LYN"}
   :DARK "ORKO"})
;{:schema [:map {:closed true} [:xy [:map {:closed true} [:x int?] [:y int?]]]],
; :value {:xy {:x 1, :y 2, :EVIL "LYN"}, :DARK "ORKO"},
; :errors (#Error{:path [2 1],
;                 :in [:xy :EVIL],
;                 :schema [:map {:closed true} [:x int?] [:y int?]],
;                 :type :malli.core/extra-key}
;           #Error{:path [],
;                  :in [:DARK],
;                  :schema [:map {:closed true} [:xy [:map {:closed true} [:x int?] [:y int?]]]],
;                  :type :malli.core/extra-key})}
#2020-12-3112:45ikitommithat should contain all the needed info to get spell-spec grade error messages.#2020-12-3112:46ikitommie.g. :in points to the actual key and :type can be used to format suggestions on misspelled keys etc.#2020-12-3113:11ikitommihttps://github.com/metosin/malli/pull/156#2020-12-3113:28ikitommialso:
(-> [:map {:closed true} [:x int?]]
    (m/explain {:x 1, :extra "key"})
    (me/humanize))
; => {:extra ["disallowed key"]}
#2020-01-0208:52ikitommiProgramming with Schemas: https://github.com/metosin/malli/pull/158#2020-01-0208:55ikitommiQuite happy how simple Schema transformations are with the current design, recursively closing all Schemas:
(defn closed-schema
  "Closes all :map Schemas recursively."
  [schema]
  (m/accept
    schema
    (m/schema-visitor
      (fn [schema]
        (if (= :map (m/name schema))
          (update-properties schema assoc :closed true)
          schema)))))
#2020-01-0313:12ikitommicomments welcome on handling extra keys with maps: https://github.com/metosin/malli/issues/43#2020-01-0515:28roklenarcicthere;’s a lot of branches in repo that have been merged and not deleted#2020-01-0521:07ikitommisure there is, I guess there is no automation removing those after merge/rebase.#2020-01-0611:20eskoshttps://help.github.com/en/github/administering-a-repository/managing-the-automatic-deletion-of-branches#2020-01-0611:20eskosYou do need to delete the old merged branches manually though; Google will find you some bash scripts for that.#2020-01-0521:09ikitommiwould like to break thing a bit: mt/strip-extra-keys-transformer should be a function of options => Transformer , not a Transformer. Question: should it remove extra keys by default: a) from all maps (open or closed) b) only from closed maps#2020-01-0521:09ikitommiwill add a option to control this, but leaning on a) as the default.#2020-01-0522:56roklenarcichm… how do I make a transformer that uses a schema property as param to transform function#2020-01-0522:57roklenarcicyou have this showcased in the :compile option example#2020-01-0522:58roklenarcicbut I’d like to look up something like that `
:math/multiplier
in the default transformer function
#2020-01-0522:59roklenarcice.g. I have datetime schema and it’s transformer would like to check if schema had :format property#2020-01-0523:00roklenarcicthe :compile thing requires one to add it to every use-site#2020-01-0523:02roklenarcicoh, I see how it’s done elsewhere, ignore that#2020-01-0611:10ikitommiyeah, the default-value-transformer uses schema properties and mounts for all schemas.#2020-01-0618:26ikitommiBreaking transformer api, hopefully making it better: https://github.com/metosin/malli/pull/162#2020-01-0618:26ikitommicomments welcome on that#2020-01-0618:29ikitommiThe goal is to have a unified api for creating Transformers: all are created via a function, instead of mixing functions and actual Transformer instances. So, instead of:
(mt/transformer
  mt/string-transformer
  mt/strip-extra-keys-transformer
  (mt/key-transformer
    {:decode #(-> % name (str "_key") keyword)
     :encode #(-> % name (str "_key"))}))
we should do:
(mt/transformer
  (mt/string-transformer)
  (mt/strip-extra-keys-transformer)
  (mt/key-transformer
    {:decode #(-> % name (str "_key") keyword)
     :encode #(-> % name (str "_key"))}))
… both work thou, thanks to auto-coercion of functions => Transformers.
#2020-01-0618:32ikitommiin other libs (schema, spec-tools) the json-transformer|matcher are just values, so people most likely try that instead of call the function. Not 100% happy with having the auto-coercion, but here thought the developer experience is more important than having an one explicit (non-familiar?) syntax.#2020-01-0812:05ikitommimerged into master#2020-01-0912:08ikitommihei, someone suggested that malli (and reitit) should be sponsored by Clojurists Together, any insight what should be developed? Thanks anyway 🙇 https://www.clojuriststogether.org/news/q1-2020-survey-results/#2020-01-0917:21roklenarcicare you sure -composite-schema function -transformer works correctly? other schemas with children use -chain helper to make sure that :leave transformers are combined in reverse order than :enter , but this one does not#2020-01-0917:35ikitommibug? PR welcome.#2020-01-1109:10ikitommiUpcoming:
(-> (m/explain
      [:map {:closed true}
       [:orders boolean?]
       [:deliver boolean?]]
      {:orders true
       :deliverz true})
    (me/with-suggestions)
    (me/humanize))
;{:deliver ["missing required key"]
; :deliverz ["likely misspelling of :deliver"]}
#2020-01-1120:46roklenarcicAdded a couple of schemas I’m using to a public repo https://github.com/RokLenarcic/malli-schemas#2020-01-1211:08ikitommiNice, date-support is on malli backlog, need to figure out a way to make it work with cljs too.#2020-01-1215:13roklenarcicGenerally, transform operations do not validate that input passes the validator first, which is good for performance, but when you have an operation like :or how do you know which branch of OR to use for transformation? Currently you try every branch as transformation and return the first result that changes the value.#2020-01-1215:13roklenarcichowever this leads to this:#2020-01-1215:14roklenarcic
(m/encode [:or
           [string? {:encode/string #(.toUpperCase %)}]
           [int? {:encode/string str}]]
          1
          mt/string-transformer)
#2020-01-1215:14roklenarcic
Execution error (IllegalArgumentException) at com.github.roklenarcic.malli-inline/eval17000$fn (form-init15839185010501867114.clj:2).
#2020-01-1215:14roklenarcic
No matching field found: toUpperCase for class java.lang.Long
#2020-01-1215:16roklenarcicnone of the built-in transformers will validate input and return the value itself if not the right type, which would make :or work correctly in this regard#2020-01-1215:28roklenarcicThe only realistic way to fix this would be to have :or do validate for each branch until it finds one that validates and then run that transform. This would perform as expected, I can prepare a PR.#2020-01-1218:48ikitommi@roklenarcic sounds right to me to select branch based on validate#2020-01-1218:49roklenarcicanother possible bug#2020-01-1218:49roklenarcicin about 5 different explainer functions in malli.core you have distance (if (seq properties) 2 1)#2020-01-1218:50roklenarcicif properties is missing then distance should be 1#2020-01-1218:50roklenarcicthe problem is that if someone puts {} as properties, then (seq properties) is false#2020-01-1218:51roklenarcicand it will say distance 1, when the form has [symbol {} children]#2020-01-1218:51ikitommitrue#2020-01-1218:51roklenarcicso the path is possibly wrong… I don’t know what the semantics of path is exactly#2020-01-1218:52ikitommiget-in to m/form of the schema with the error :in should point to schema#2020-01-1218:52ikitommineeded in pretty error reporting , which is wip.#2020-01-1218:52roklenarcicbut you see what I’m getting at… (seq properties) is nil for nil and {}, not good for counting elements in form#2020-01-1218:53ikitommiyes, that's currently wrong but should be easy to fix#2020-01-1218:53ikitommiGood thing that it's pre-alpha ;)#2020-01-1218:54ikitommiWill need malli at a project starting tomorrow, so good reason to push out first alpha too#2020-01-1220:19ikitommishould the schemas allow nil children? e.g. [:and int? nil pos-int?]. Currently :map allows, composite schemas don’t.#2020-01-1220:20ikitommie.g. [:map [:x int?] nil [:y pos-int?]] works, [:and int? nil pos-int?] doesn’t.#2020-01-1220:20ikitommiall core schemas should behave identically.#2020-01-1220:22ikitommialso, currently:
(m/form [:and {} int?])
; => [:and int?]

(m/properties [:and {} int?])
; => {}
#2020-01-1220:23ikitommishould empty properties be removed (set to nil) already when schema is created from AST?#2020-01-1220:23ikitommiafter that:
(m/form [:and {} int?])
; => [:and int?]
(m/properties [:and {} int?])
; => nil
#2020-01-1220:28ikitommimy suggestion: • allow nil child schemas in all core schemas: easy to create programmatically (e.g. [:map [:x int?] (if require-y [:y int?])]) • treat empty properties as nil, allows (`[:map (if close-maps {:closed true}) [:x int?]]`)#2020-01-1220:38roklenarcicsure, sounds sensible#2020-01-1222:00roklenarciccan explainer fn return null? I thought it must return acc#2020-01-1223:09pithylessI agree all core schemas should behave identically; I would suggest not ignoring nil children, which gives more obvious semantics in cases like this: [:and int? nil pos-int?] (this would always be false, instead of implementing a special-case for nil that behaves differently than regular "falsey" semantics in Clojure). In the map example, I find it more intuitive to use malli.util/merge semantics to describe programmmatic optionality: (malli.util/merge [:map [:x int?]] (if require-y [:map [:y int?]]))#2020-01-1305:45ikitommifixed so that nil is allowed as properties and empty properties are treated as nil (and stripped from m/form).#2020-01-1305:51ikitommihttps://github.com/metosin/malli/pull/164#2020-01-1305:46ikitommiI don’t think nil should be treated as “always fail” in composite schemas. Currently, it will blow as nil is not a valid schema.#2020-01-1305:47ikitommialso, can’t strip all nils from children, as things like :enum might have actual valid value of nil.#2020-01-1305:49ikitommiSo, options being: 1. allow (and strip) nil in map & composite schemas, for convenience sake 2. blow up in case of nil as child schema 3. map nil to “always fail”#2020-01-1305:50ikitommithe malli.util merge is not that fun for deeply nested maps:
(malli.util/merge 
  [:map [:x [:map [:y [:map [:z int?]] 
  (if require-z2 [:map [:x [:map [:y [:map [:z2 int?]]))
#2020-01-1308:54roklenarciceach into-schema should probably decide for itself what to do with nil values#2020-01-1414:53ikitommimerged the spell-checking, https://twitter.com/ikitommi/status/1217096807316119553#2020-01-1520:06rschmuklerHey @ikitommi just a heads up that https://github.com/teknql/pablo exists for when / if you go to break malli into sperate libraries. It's still early days, but it's made to ease the burden of managing mono-repo-like libraries. Documentation is sparse at the moment, but I don't anticipate that you'll be slicing malli up too soon either#2020-01-1520:36ikitommiOh, that looks great! How do you setup the artifact version when pushing to clojars?#2020-01-1521:14rschmuklerTwo options (again, sorry on the sparse docs)#2020-01-1521:14rschmuklereither, you invoke it with an explicit :version via the repl / CLI (cli coming soon)#2020-01-1521:14rschmukleror, it automatically looks at your git status and uses the tag#2020-01-1521:15rschmuklerie. if git is tagged at 1.3.0 and not dirty, it's 1.3.0, if it's dirty then it's 1.3.0-SNAPSHOT#2020-01-1521:15miikkaHmh #2020-01-1521:15rschmuklerMight also make it so that if you're ahead of a tag (but not dirty) it does 1.3.0-SNAPSHOT too#2020-01-1521:15miikkaWhy not just use Leiningen?#2020-01-1521:16rschmuklerI ask myself that at least once a week 😉#2020-01-1521:17rschmuklerI'm hoping to get pablo to ultimately compile as a native image#2020-01-1521:18rschmuklerAlso, I think the biggest thing that tools-deps gets right is that it forces your project into a edn map - lein allows real programming to happen inside the project.clj file which can allow it to become quite the mess#2020-01-1521:19rschmuklerBut, I don't disagree that this is shoe-horning some lein-like functionality into tools deps#2020-01-1521:21miikkaYeah… my ideal setup right now would be actually to have Leiningen for the project management (because it already works) and tools.deps for dependency management. Too bad that lein-tools-deps doesn’t quite make it work. #2020-01-1521:23rschmuklerThe thing that kills me with tools-deps is that they didn't make a project.clj parser for when cloning via git... So if you attempt to depend on a git repo and they use lein and don't publish the xml file in the repo, you're SOL#2020-01-1521:24miikkaYeah#2020-01-1521:24rschmuklerIt feels very unpragmatic#2020-01-1521:24miikkaWell, you can’t correctly parse project.clj without running it. Although of course you could make it work in most cases. #2020-01-1521:25rschmuklerYeah, 1) you could naively parse it and cover 99%, and 2) sci now exists - so you could eval it at make it work in 99.9999% of cases#2020-01-1521:26miikkaI think tools.deps is designed in such a way that you can add new dep types. But it’s not nice if it’s not there out of the box#2020-01-1521:27rschmuklerYeah I think you're right - I think the coordinates can all be expanded on via multi-method. But then I'm not quite sure how you get it to load those multimethods before trying to resolve the local deps.edn file#2020-01-1521:28miikkaYeah#2020-01-1716:05theeternalpulseHi folks, starting to incorporate malli into my project, was wondering is there a mechanism to add metadata to a function so that input and output are automatically validated, or is all the validation meant to be done manually within the function?#2020-01-1716:15ikitommi@theeternalpulse two choises: 1. wait for https://github.com/metosin/malli/issues/125 2. try aave, https://github.com/teknql/aave#2020-01-1716:16theeternalpulseoh sweet#2020-01-1716:16ikitommigoal of malli is to be a complete schema-system, both function schemas and linting (integration to clj-condo) will be there, soon, I hope.#2020-01-1716:17theeternalpulse👍 thanks, I'll take a look at aave in the meantime.#2020-01-1716:18ikitommiany discussion of aave most welcome here, as @rschmukler is here too.#2020-01-1716:21rschmuklerHi @theeternalpulse! Aave is in its early days but it does exist! It's syntax is extendable so if you don't like the ghostwheel-like syntax, it wouldn't be too hard to add. The other thing that I want to add is the option for compile time testing of pure functions (so we refuse to compile, rather than wait for the user to call the function). If you have any feature requirements please feel free to open some issues#2020-01-2216:36ikitommithe m/accept will need extra parameters, maybe both :in and :path. Need to be able to better support schema mappings and inferring ui’s from mall schemas.#2020-01-2216:37ikitommiwill break all 3rd party Schema extensons, sorry.#2020-02-0417:36ikitommimaybe there should be default-options-provider, that can be overridden once?#2020-02-0417:36ikitommi, would allow easy plugging in of custom registeries, and other options how things work#2020-02-0417:37ikitommi(integrating malli into real projects, need custom global schemas without a hassle)#2020-02-0510:31borkdudeMaybe metosin wants to be mentioned here because they are using sci in malli? https://github.com/borkdude/babashka/issues/254#2020-02-0510:49ikitommidid. I guess you have logo already from clj-condo question 😉#2020-02-0510:54borkdudeyep, thanks#2020-02-0618:58eoliphantwill malli eventually do, for lack of a better term, a keyset coerce/conform? I try to use open keysets as much as possible, but always struggle the best way to handle stuff like the following
(-> {:a .. b:.. :c .. :d} ; say input from api call
     validate ; we only care about :a,:b and :c but ignore :d
     do-something-with-a-and-b ; fine with a s/keys [:a :b] or m/validate equivalent precondition
     do-something-with-c ; fine with a s/keys [:c]
     (select-keys [:a ::c]) ; or (transform my-schema)
     (write-a-b-c-to-datomic) ; at this point I want a closed schema/spec precond, that could potentially narrow the keys on the way in? 
#2020-02-0619:05ikitommi@eoliphant would something like (mu/select-keys map-of-abc [:a :b]) be good? you can always make maps closed, to either fail on extra keys or to strip extra keys in transform#2020-02-0619:05ikitommie.g. select-keys, but for schemas#2020-02-0619:08eoliphanthmm yeah, perhaps strip extra on transform is the way to go? again, just been thinking about this, in terms of a structural coercion, but that might not be the best approach#2020-02-0619:13ikitommiI think it's the way to go: open schemas are great , but external systems like db's aren't always happy with extra stuff in. We are at least stripping data away at the boundaries.#2020-02-0619:13ikitommithere is malli.transform/strip-extra-keys-transformerfor this#2020-02-0619:16eoliphantespecially for stuff like datomic lets you do maps, etc all the way down, but it gets understandably upset if you pass in attributes it doesn’t know about 😉#2020-02-0720:42eoliphanthi quick question on error messages. I’m trying to do a custom message for each condition, so for your initial example, somehting like
(-> [:and
     [int? {:error/message "Not int"}]
     [:> 6 {:error/message "Less than 6"}]]
  (m/explain 6))
That doesn’t work, nor does say this [[:> 6] {:error/message "Less than 6"}]] though
(-> [:and
     [int? {:error/message "Not int"}]
     [:> 6]]
  (m/explain "a"))
is just fine
#2020-02-0806:38ikitommiproperties are always (if defined) the first argument in the schema vector, so [:> {...} 6] should work#2020-02-0806:39ikitommi"hiccup syntax"#2020-02-0814:51eskosHow battle tested is the serializable function schemas feature of malli? I'm tinkering on a small side project of mine and realized malli would actually fit into it really well for the purposes of schema+coercion of data structures saving me the effort of doing those myself... 🙂#2020-02-0815:16ikitommi@suomi.esko heavy lifting done by https://github.com/borkdude/sci#2020-02-0815:17ikitommimalli should be in alpha around ClojureD#2020-02-1016:49ikitommistarted to add programmatic helpers to malli: https://github.com/metosin/malli/pull/172#2020-02-1016:50ikitommilooks like:
(malli.util/get-in
  [:map {:title "test"}
   [:x [:vector
        [:list
         [:set
          [:sequential
           [:tuple int? [:map [:y [:maybe boolean?]]]]]]]]]]
  [:x 0 0 0 0 1 :y 0])
; => boolean?
#2020-02-1016:51ikitommithere is a new Protocol LookupSchema , which is implemented for :map, :multi, :vector, :list, :set, :sequential, :tuple.#2020-02-1016:52ikitommi
(malli.util/select-keys
  [:map {:title "map"}
   [:a int?]
   [:b {:optional true} int?]
   [:c string?]]
  [:a ::extra])
;[:map {:title "map"}
; [:a int?]
; [:b {:optional true} int?]]
#2020-02-1016:52borkdudeso kinda like the new spec2 thing, where you can select from your schema in different contexts?#2020-02-1016:54ikitommiwell, almost. like in schema-tools, going to use the clojure.core functions names. There could be malli.util/select to do things like spec2 does.#2020-02-1016:54ikitommi
(-> [:map
     [:a int?]
     [:b string?]
     [:c [:map
          [:x int?]
          [:y keyword?]]]]
    (mu/select-keys [:a :c])
    (mu/update-in [:c] mu/dissoc :y))
;[:map
; [:a int?]
; [:c [:map
;      [:x int?]]]]
#2020-02-1016:55borkdudenice#2020-02-1016:55ikitommithat’s the thing we have been using with prismatic schema for 3+ years now, happy with that 🙂#2020-02-1016:57ikitommiI’m not sure how the spec2 select works with nested sequential specs#2020-02-1016:58ikitommiin malli, I want to keep things explicit: you need to walk those over yourself.#2020-02-1016:58borkdudeme neither, I kinda lost track on spec2#2020-02-1016:58ikitommi
LookupSchema
(-get [_ key default] (if (= 0 key) schema default)))
#2020-02-1016:59ikitommi^:--- for all sequential specs, all have just the 0#2020-02-1016:59ikitommialso, planning to support schamas as maps, via a helper.#2020-02-1017:00ikitommi
[:vector {:min 0, :max 10} int?]
; =>
{:name :vector
 :properties {:min 0, :max 10}
 :children [int?]}
#2020-02-1017:01ikitommithere is already a walker that recursively transforms between formats.#2020-02-1017:02ikitommiwill push these extras into malli.util, so there is just the absolute minimum stuff in the core.#2020-02-1021:47ikitommishould the m/name be m/type? In both React & json schema it's a type#2020-02-1021:48ikitommi
[:vector {:min 0, :max 10} int?]
; =>
{:type :vector
 :properties {:min 0, :max 10}
 :children [int?]}
#2020-02-1106:06ikitommi
(mu/assoc-in nil [:a ::c :d] int?)
; [:map [:a [:map [:b [:map [:c [:map [:d int?]]]]]]]]
#2020-02-1106:08ikitommigreat thing about clojure is the functions compose nicely. HOFs are mostly 1:1 counterparts from Clojure (just cleaned up):
(defn update-in
  "Like [[clojure.core/update-in]], but for LensSchemas."
  [schema ks f & args]
  (letfn [(up [s [k & ks] f args]
            (assoc s k (if ks (up (get s k) ks f args)
                              (apply f (get s k) args))))]
    (up schema ks f args)))
#2020-02-1106:09ikitommi
(defn update
  "Like [[clojure.core/update]], but for LensSchemas."
  [schema key f & args]
  (let [schema (m/schema schema)]
    (m/-set schema key (apply f (m/-get schema key nil) args))))
#2020-02-1106:09ikitommiInternally, there are just -get and -set functions in a LensSchema protocol. Dead simple.#2020-02-1303:02eckardjfHow could I use a custom error message for a failed regex match? Something like:
(-> [:map [:a [#"^[0-9]+$" {:error/message "should match"}]]]
    (mc/explain {:a "aaa"})
    (me/humanize))
#2020-02-1305:29eckardjfAh, I see what I was missing - this works
[:map [:a [:re {:error/message "should match"} #"^[0-9]+$"]]]
#2020-02-1408:02ikitommiI'm bit surprised that the first one doesn't work.#2020-02-1414:20plexusI'm trying to use malli to coerce reitit query arguments, but not quite getting the results I'm expecting.#2020-02-1414:21plexusroute:
["/paginated/:table"
 {:get {:coercion reitit.coercion.malli/coercion
        :parameters {:path {:table string?}
                     :query {:page int?
                             :page-size int?}}
        :responses {200 {:body any?}}
        :handler (fn [{:keys [path-params params] :as req}]
                   {:status 200
                    :body "OK"})}}]
#2020-02-1414:22plexusand router#2020-02-1414:22plexus
(http/router routes {:data {:middleware [rrc/coerce-exceptions-middleware
                                         rrc/coerce-request-middleware
                                         rrc/coerce-response-middleware]}})
#2020-02-1414:22plexusis it reasonable to expect that this way a ?page=1 parameter will come in as {:query-params {"page" 1}}?#2020-02-1414:25ikitommiyes, but the coerced params are under :parameters, so_
{:parameters {:query {{:page 1}}}
#2020-02-1414:22ikitommi
{:table string?} --> [:map [:table string?]]
#2020-02-1414:22plexusand not {"page" "1"}#2020-02-1414:23plexusahhh thanks, let me try that#2020-02-1414:23plexussame for :query then I suppose? [:map {:page ...}]#2020-02-1414:23ikitommididn’t add any sugar for the top-level maps in the malli coercion, so, all parameters need to be defined in the malli syntax.#2020-02-1414:24ikitommicrossed my mind to do that, would be 1:1 to switch the simple cases from data-specs to malli…#2020-02-1414:25plexusstill getting string...
:parameters {:path [:map
                               [:table string?]]
                        :query [:map
                                [:page int?]
                                [:page-size int?]]}
#2020-02-1414:26ikitommithere is an example app in https://github.com/metosin/reitit/tree/master/examples/ring-malli-swagger#2020-02-1414:27ikitommi
["/math"
        {:swagger {:tags ["math"]}}

        ["/plus"
         {:get {:summary "plus with malli query parameters"
                :parameters {:query [:map [:x int?] [:y int?]]}
                :responses {200 {:body [:map [:total int?]]}}
                :handler (fn [{{{:keys [x y]} :query} :parameters}]
                           {:status 200
                            :body {:total (+ x y)}})}
          :post {:summary "plus with malli body parameters"
                 :parameters {:body [:map [:x int?] [:y int?]]}
                 :responses {200 {:body [:map [:total int?]]}}
                 :handler (fn [{{{:keys [x y]} :body} :parameters}]
                            {:status 200
                             :body {:total (+ x y)}})}}]]]
#2020-02-1414:29plexussorry, had some connectivity issues there. Let me have a look at the example.#2020-02-1414:32plexusI guess you use all the reitit.*.middleware instead of ring-defaults now#2020-02-1414:45ikitommithere are useful middeware there, but for example the site is really heavy and many things are solved alread (like fallback to look up file resources)#2020-02-1414:45plexusok, progress. changed reitit.http/router to reitit.ring/router, and now I'm getting a 406 Not Acceptable#2020-02-1414:46ikitommithere is a bug somewhere in the malli-coercion, bumped into it few days ago, if the response is not valid, gives really weird error. will fix that soon.#2020-02-1414:47plexusadded Accept-Encoding=application/json and now it's a 405 Method Not Allowed#2020-02-1414:48plexusstill going to consider this progress 🙂#2020-02-1414:50ikitommiI have a minimal shadow-cljs + deps + reitit + malli example project, will good defaults, try to push that out before ClojureD#2020-02-1414:51plexusthat'd be great, thanks!#2020-02-1414:51plexuswas hoping not to manually have to parse those integers but maybe the trouble isn't worth it right now#2020-02-1414:51ikitommidefault reitit has no opinions, lot’s of things that need to decided, not the friendliest for new users#2020-02-1414:52ikitommiif you test the sample app I pointed and just strip away things?#2020-02-1414:52ikitommithere is also the request/response printer, just commented out.#2020-02-1414:52ikitommilet’s you see what the different mw do in the chain.#2020-02-1414:53ikitommiuses deep-diff btw 😉#2020-02-1414:55plexusnice, probably why it's been doing so well. most downloaded lambdaisland project 🙂#2020-02-1414:59plexuswe're working on clojurescript support for deep-diff BTW#2020-02-1421:36steveb8n@U07FP7QJ0 this is great news. when using Shadow-cljs and Cursive, there’s basically no easy visual diff tooling that I can find. cljs support in deep-diff will fill this gap perfectly#2020-02-1415:02plexusoh it's working now!#2020-02-1415:03plexusmisunderstood something about httpie :face_palm::skin-tone-2:#2020-02-1415:18ikitommiIt looks like malli need support for sequence schemas. I just need the simplest “varargs” case, why not do the whole thing while at it.#2020-02-1415:18ikitommiI’m thinking of re-using the map/multi syntax:
(m/valid?
  [:cat
   [:x int?]
   [:y int?]
   [:rest [:* string?]]]
  [1 2 "kikka" "kukka"])
; => true
#2020-02-1415:19ikitommiif https://github.com/cgrand/seqexp would be ported to cljs, could use that behind the scenes.#2020-02-1415:21ikitommilike clojure.spec.alpha/conform, there could be something for that?
(m/destructure
  [:cat
   [:x int?]
   [:y int?]
   [:rest [:* string?]]]
  [1 2 "kikka" "kukka"])
; {:x 1
;  :y 2
;  :rest ("kikka "kukka")}
#2020-02-1415:22ikitommiwithout that, this would be just enough:
(m/valid?
  [:cat int? int? [:* string?]]
  [1 2 "kikka" "kukka"])
; => true
#2020-02-1415:23ikitommiideas most welcome on this.#2020-02-1415:31ikitommi
(require '[net.cgrand.seqexp :as se])

(se/exec
  (se/cat
    (se/as :x int?)
    (se/as :y int?)
    (se/as :restz (se/* :string?)))
  [1 2 "kikka" "kukka"])
;{:x (1)
; :y (2)
; :restz ("kikka" "kukka")
; :rest ()}
#2020-02-1419:07borkdude@ikitommi fwiw, I ported clojure spec to a graalvm compatible subset here: https://github.com/borkdude/spartan.spec#2020-02-1419:07borkdudeit contains all the regex ops#2020-02-1419:35ikitommioh, nice. Will have a look.#2020-02-1419:36borkdudeit's basically a copy with things removed#2020-02-1419:38ikitommiWhat if explainer would also report succesfully validated values in same format as errors? At top level, the m/explain it would return either errors or succefull values.#2020-02-1419:38ikitommiit would give destructuring basically for free.#2020-02-1419:41ikitomminow, only errors are pushed to the accumulator. It could be swapped Into a protocol, which has -explain-error and -explain-success functions. Schemas write to those and internally, in case of first error, it just tosses away the successes and collects only errors. In case of no errors, it would return the successes in the same explain -format!!#2020-02-1419:44ikitommias it contains both the paths to schemas and in data, one can "unexplain" in a generic way#2020-02-1419:57ikitommiSpec has conform, unform and explain , malli could only have explain for all these, but has validate for perf reasons. Still, less for more.#2020-02-1420:02borkdudein spec the bulk of the work is done in conform. this result is fed into explain which transforms that result to something readable, but even valid? uses conform#2020-02-1509:56ikitommiI don’t think result of s/conform is fed into explain. It either returns the conformed value of ::s/invalid. In case of latter, s/explain is called again with the original data. Which is fine, as it’s the failure path as you said.#2020-02-1509:57ikitommim/validate does much less than s/valid? and because of that - is much faster.#2020-02-1509:58ikitommior, actually, m/validator.#2020-02-1509:58ikitommi
;; 40ns
(let [spec (s/and int? (s/or :pos-int pos-int? :neg-int neg-int?))
      valid? (partial s/valid? spec)]
  (cc/quick-bench
    (valid? 0)))

;; 5ns
(let [valid? (m/validator [:and int? [:or pos-int? neg-int?]])]
  (cc/quick-bench
    (valid? 0)))
#2020-02-1420:02borkdudeI think might have the philosophy that when something is valid it should go fast, but when something is invalid, it doesn't matter if you have to call conform again#2020-02-1420:03borkdudebecause something is wrong anyway#2020-02-1511:07ikitommino unform in spartan-spec?#2020-02-1512:41borkdudesince I've never used it, I didn't port it yet 😛#2020-02-1512:41borkdudePR welcome!#2020-02-1917:03ikitommi
(require '[malli.core :as m])
(require '[malli.transform :as mt])

(def transformer
  (mt/transformer
    ;; first run schema-based transformations named :before
    {:name :before}
    ;; run json-things
    (mt/json-transformer)
    ;; add default values
    (mt/default-value-transformer)
    ;; run custom things
    {:name :after}))

(m/decoder
  [:map
   [:name string?]
   [:age int?]
   [:address
    [:map
     [:street string?]
     [:country [:enum "finland" "germany"]]]]]
  transformer)
; => clojure.core$identity
#2020-02-1917:04ikitommiI keep surprised how good the transformation engine really is.#2020-02-2414:01ScarHey everyone. Love using this tool. Seems like validation works blazing fast, however mp/provide is slow. for example - creating a schema from a vector of 200 items will take more than 2 minutes. The processing growth is linear so 20,000 would take about 9 hours :(. I think it has to do with the fact the schema is built using exceptions.#2020-02-2414:03Scar#2020-02-2417:45ikitommioh, that's horrible. The only really perf optimized trails are -validator, -explainer and -transformer paths of schemas. Ideas welcome how to make the providers faster.#2020-02-2417:47ikitommithe algo is currently brute force, but I think if there were some kind of type-based narrower, it would not throw. e.g. Java Long would only be checked against predicates that work with that.#2020-02-2508:46Ben Slessoff the top of my head, wouldn't "errors as values" be faster? i.e. instead of throwing, return the data in a map with an :error key#2020-02-2515:39ikitommiErrors as values would be slower in the happy case / success - one would have to check always is the result an error or not. Try is basically zero cost in JVM, so the perf penalty (of throwing) only happens at the unhappy path.#2020-02-2515:41ikitommiI think the exceptions originate from the fact that the inferrer tries to make all registered schemas from it and they throw on invalid childs. E.g. for value 1, it tries to make a map with [:map 1], which throws on creation.#2020-02-2515:42ikitomminarrowing possible schemas to test would make it orders of magnitude faster, my guess on 2 (orders)#2020-02-2612:55eskosThere’s IIRC some newer optimizations in post-JDK9 versions for making stack&exception handling way faster, but I haven’t really looked into those beyond StackWalker…at least one of them was a throw-without-resolving-stack kind of operation; the stack unrolling is what generally is the slow part when exception gets thrown.#2020-02-2612:57eskosIt is also maybe possible to memoize the exception itself, although I wonder if that would work at all with the ex-info…#2020-02-2812:36ikitommijust in time for tomorrows talk 🙂#2020-02-2812:37ikitommithat’s clj-kondo talking#2020-02-2823:44JorinLooking forward!#2020-02-2908:55kszabogood luck with the talk!#2020-02-2911:27ikitommithanks! the slides are here: https://www.slideshare.net/mobile/metosin/malli-inside-datadriven-schemas#2020-03-0207:44eskosIs there going to be a video of this?#2020-02-2912:27ziltiNice, thanks! I enjoyed the presentation. Considering the library for the project at work now.#2020-02-2912:29ziltiYou mentioned something about generating input forms, right?#2020-03-0214:39ikitommiwe have been doing that is many projects, but with other schema-libs, nothing library-quality for malli yet. Did a spike using antd+malli, and learned that we need to add 1-2 extra args to m/-accept for this.#2020-03-0214:40ikitommionce it’s done, goal is to make an alternative schema backend for https://domino-clj.github.io/ from malli.#2020-03-0300:38ziltiI see. Thing is I am very interested in everything that gives me a bit of a hint for how to tacke flexible form generation for a project at work. Do you happen to have a link to something, no matter if done with another schema lib?#2020-02-2916:04Vincent CantinIs there some guidelines for contributing to Malli?#2020-03-0214:40ikitomminot yet, should be,.#2020-03-0214:40ikitomminot yet, should be,.#2020-03-0204:57Vincent Cantinif I understand correctly, the most natural way to implement the equivalent of spec/conform and spec/unform in Malli is to use a Transformer, isn't it?#2020-03-0214:42ikitommiare you interested in the branching information from s/conform?#2020-03-0214:43Vincent Cantinyes, totally. I need it for my Vrac project.#2020-03-0214:44ikitommiCurrently, there is no such thing. But one idea was that the m/explain also collected the explain info from the successfully validated values.#2020-03-0214:45ikitommi… that would contain all the needed branching information and could be used to produce “humanized” result like the one s/conform does.#2020-03-0214:46ikitommialso, would be easy to “unform” that as all the locations and values already exist there.#2020-03-0214:50Vincent CantinI am not totally convinced that the function explain should also be used for s/conform. The implementation might be similar, but explain and conform have different purposes.#2020-03-0214:51Vincent CantinThat’s why I was wondering if it would be better to have a separate function for that, maybe via a Transformer.#2020-03-0214:52Vincent CantinI did not finish to play with Malli, not sure if the Transformer pattern is suitable for those 2 functions.#2020-03-0214:55ikitommiyou should be able to do that with a transformer#2020-03-0209:32borkdudeEnjoyed the presentation, thanks!#2020-03-0210:24orestisI browsed the slides - is the Kondo integration a vision for the future or does it work today already? I’m super excited!#2020-03-0215:02ikitommiit was a 15min hack to check if that would work and it did! I just emitted a lint.edn which the fn description in the form clj-kondo understands it and hard-wired clj-kondo to read that file. I think there is betters ways to do this automatically. The m/defn is also not finished, but will be a 1:1 port from plumatic schema so cursive knows that too. Might take few (dev) days to get them cleaned up and pushed to master for testing.#2020-03-0218:00orestisSuper cool! I’ve wanted to generate specs for GraphQL input objects which are closed maps, and being able to statically analyze functions that consume them. Looks like this might be possible soon!#2020-03-0218:02orestisOh and I could probably also annotate React components that consume GraphQL...#2020-03-0213:03teodorluThanks for the great talk on ClojureD, @ikitommi!#2020-03-0213:05teodorluI stumbled over an article that I figured people here might find interesting: QuickREST: Property-based Test Generation of OpenAPI-Described RESTful APIs The article discusses generating Clojure specs from OpenAPI specifications, and running generative testing against those. With Malli, I figure that process might become even simpler. https://arxiv.org/pdf/1912.09686.pdf#2020-03-0214:56ikitommiConform/unform using explain would look something like this:
(def Schema
  [:map
   [:a [:or pos-int? string?]]
   [:b string?]])

(m/explain Schema {:a 42, :"a"})
;{:schema [:map
;          [:a [:or pos-int? string?]]
;          [:b string?]]
; :value {:a 42, :"a"},
; :results [#Success{:path [1 1 1], :in [:a], :schema pos-int?, :value 42}
;           #Success{:path [2 1], :in [:b], :schema string?, :value "a"}]}

(-> Schema
    (m/explain {:a 42 :b "a"})
    (me/humanize))
;{:a #Branch{1 42}
; :b "a"}
#2020-03-0215:03Vincent Cantinin my use case with spec, I encountered a few times the need to apply custom transformations during s/conform , that’s also why I would think that a Transformer would be better if it can be easily customized to produce the format I want in one pass.#2020-03-0215:04ikitommi@teodorlu looks interesting. need to read that when extra time.#2020-03-0215:05ikitommiyeah, transformers give you the single pass. looking forward to seeing if that helps over spec.#2020-03-0217:18ikitommi@borkdude the :keys doesn’t work with :ret values with clj-kondo yet?#2020-03-0217:18ikitommion :args it seems to work, which is awesome 🙂#2020-03-0217:19borkdude@ikitommi what :ret type are you using?#2020-03-0217:21ikitommi
{:args [{:op :keys
         :req {:a {:op :keys
                   :req {:b {:op :keys
                             :req {:c :int}}}}}}]
 :ret {:op :keys
       :req {:b {:op :keys
                 :req {:c :int}}}}}
#2020-03-0217:22borkdudeI don't think that's supported yet. As a fallback you can use :tag :map for now and post an issue about it.#2020-03-0217:22borkdudemaybe it works when you add :tag :map in that same map you pass to :ret now, so you can keep the code the same when it gets fixed?#2020-03-0217:25ikitommiwill write an issue, the :tag :map makes it a map, but the keys are not followed, at least (:a my-fn) succeed despite the ret says there is no such key#2020-03-0217:25borkdudethanks#2020-03-0217:59ikitommikinda bad explanation, but https://github.com/borkdude/clj-kondo/issues/783#2020-03-0415:56plexusI vaguely remember talk at ClojuTRE about generating forms / UI based on Malli. Has anyone tried something like that? Any interesting pointers?#2020-03-0416:53ziltiYea that is something I am highly interested in finding out#2020-03-0418:37ikitommigenerating UIs - there are many ways to do that: 1. walk the malli schema and emit an ui-application out of that, did a quick hack of that in a project, reagent + antd 2. walk the malli schema and emit an ui-schema out of that, to be rendered somehow 3. have both malli schema (“the model”) and an ui-schema (“the layout”) and use them together to generate the ui#2020-03-0418:39ikitommiin 1 & 2, the malli.visitor needs 1-2 extra args to be usefull: all walked schemas should know their place in the schema (e.g. the :in in explain) and in the data (the :path). This enables the ui-elements to be standalone, the ui-component knows where in the app-state the data needs to be stored.#2020-03-0418:40ikitommiI used formik-style custom helper to get the usability-stuff done right, dirty form state etc.#2020-03-0418:41ikitommibut, nothing Ready now, if someone has time and skills at making awesome forms, happy to help with that.#2020-03-0418:42ikitommithere is https://domino-clj.github.io/, idea was that it’s own model could be swapped with malli (just need a protocol to abstract the model and make malli-domino wrapper). It has also a graph-engine to calculate values & other cool features.#2020-03-0418:45ikitommichecked the js-side form generators last year, some good things to learn from there, e.g. https://github.com/rjsf-team/react-jsonschema-form & https://jsonforms.io/#2020-03-0418:51ikitommiin my ad-hoc generator, had ui-hints as properties like this: [:and {:ui/label "arvostelu", :ui/type :rate} int?]#2020-03-0418:52ikitommiemitting a https://ant.design/components/rate/#2020-03-0506:37ikitommihttps://github.com/metosin/malli/pull/184 <-- adding the :in to visitor, :path not needed.#2020-03-0506:38ikitommifxed :path with :multi schema: https://github.com/metosin/malli/pull/185#2020-03-0506:39ikitommi#2020-03-0506:40ikitommias with sequencial schemas, we don’t know the actual index of the value, so added just a placeholder “value here” using :malli.core/in marker. Comments welcome on that.#2020-03-0506:44ikitommianother options would be to: 1. use -1, e.g. temp-id like in datomic 2. use 0, the first index#2020-03-0507:57eskosre: Form generation based on malli discussed yesterday, I’ve been thinking that since malli has the capability, it could be interesting to provide the entire form with the validator functions from backend - in projects I’ve been eg. translations for error messages are managed on the backend services anyway, so generating a full-blown “form scheme” with not just what the desired data structure is but also how it’s validated and what the translations and help texts, labels etc. are in various languages and whatever else could actually be the most interesting usability wise. This approach would of course mean doing some extra heavy lifting as binding this data would mean the UI component (think React) would need to have indirection to the path to data (both read and updates) instead of being instanced directly with the data, but that is a better architecture anyway. This was discussed in some conference last year, I’ll try to find a link…#2020-03-0508:21eskosFor the life of me I can’t find the post/talk. Anyway it was mostly just a suggestion that 1. have global state (eg. re-frame’s db) 2. all components should know only the path to their data in the state (`[:foo :went :to :bar]`) 3. all updates to data is done to that same path What my caffeinated flow of mind above tried to convey was that with this pattern having the malli data to be mounted to some path in semi-generic fashion would allow instancing the form components quite nicely as well 🙂
#2020-03-0508:24ikitommidid you check my code snipplet from above? allows to add the path to data…#2020-03-0508:25ikitommiui-components need to know how to handle sequences and :map-ofs, e.g. need to generate / retain indexes for the elementes. for nested maps & tuples, the path can be fully calculated forehand.#2020-03-0508:36eskosYes, now that I re-read that I think we’re at least in the same ballpark with how it should be.#2020-03-0508:37eskosgot me thinking, does malli handle seq keys for maps? 😅 there’s this one terrible API I saw recently which returned data like this {[:foo :bar] :baz [:more] :stuff}#2020-03-0508:52ikitommiwhat other ways there are to handle seq keys than to use seqs like in the example?#2020-03-0508:53eskosThe first step is to make sure that is even parsed correctly 🙂 afaik python/ruby/php can’t even read that in#2020-03-0508:54eskos(why am I on the weird side of this industry all the time…)#2020-03-0508:57eskosAnyway, the biggest concern with that is that correct validator is matched per key which actually a sequence; in my laziness I made that example uniform while in practice it probably wouldn’t be.#2020-03-0610:02teodorluJust pasted a huge JSON sequence into https://malli.io and got a nice, readable schema out, with optional values, sequences and everything in place. What a pleasure! Thanks! 🙏#2020-03-0610:04mike_ananevHi! I'm trying to mu/merge two specs [:map ...] and [:multi ... ]. Why I always have only one of them in a result? Is there any examples?#2020-03-0610:08ikitommi@mike1452 they are of different types and the last one wins (like with clojure.core/merge)#2020-03-0610:09mike_ananev@ikitommi thanx.#2020-03-0610:09ikitommibut, the merge is far from perfect, not sure what happens if one merges two multis for example.#2020-03-0610:10ikitommii would assume that they get merged together, but not sure if the impl has a special condition just for :map (should be for MapSchemas, which both multi and map are)#2020-03-0610:11ikitommi… with that change, could actually merge those togerher. not sure if that’s a good idea thou :thinking_face:#2020-03-0610:11mike_ananevanother question: when I generate multi spec, why I have random values in dispatch type instead of those values that in multi spec?
(def mumap
  [:multi {:dispatch :db/shared-channel}
   ["folder" [:map [:db/shared-channel string?] [:db/shared-channel-folder string?]]]
   ["web" [:map [:db/shared-channel string?] [:db/shared-channel-url string?]]]])
#2020-03-0610:12mike_ananevinstead of "folder" and "web" in shared channel i have random values. I expect only these two values#2020-03-0610:12ikitommithe generator is picked from the multi branch, which doesn’t limit the key.#2020-03-0610:12ikitommitry:
[:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]]
#2020-03-0610:14mike_ananevIt helps, thanks!#2020-03-0610:14ikitommias the dispatch can be anything, it’s non-trivial to use the branch name in a generator automatically#2020-03-0610:39mike_ananevI have data structure in which first half is always the same and second half is multi-spec based on :db/shared-channel value
(def sample1 {:db/type                  "h2",
              :h2/user                  "sa",
              :h2/filename              "./boxdb",

              :db/shared-channel        "folder",
              :db/shared-channel-folder "dev/resources/shared"})

(def sample2 {:db/type               "h2",
              :h2/user               "sa",
              :h2/filename           "./boxdb",

              :db/shared-channel     "web",
              :db/shared-channel-url ""})
#2020-03-0610:39mike_ananevI did multi spec
(def mumap
  [:multi {:dispatch :db/shared-channel}
   ["folder" [:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]
              [:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
              [:h2/filename {:default "./boxdb"} string?]
              [:h2/user {:default "sa"} string?]]]
   ["web" [:map [:db/shared-channel [:= "web"]] [:db/shared-channel-url string?]
           [:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
           [:h2/filename {:default "./boxdb"} string?]
           [:h2/user {:default "sa"} string?]]]])
#2020-03-0610:40mike_ananevQuestion: how to avoid duplication of first half? (merge is not work like in spec1)#2020-03-0610:56mike_ananevI need something like:
[:map
 [:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
 [:h2/filename {:default "./boxdb"} string?]
 [:h2/user {:default "sa"} string?]
 [:multi {:dispatch :db/shared-channel}
  ["folder" [:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]]]
  ["web" [:map [:db/shared-channel [:= "web"]] [:db/shared-channel-web string?]]]]]
#2020-03-0610:59ikitommimany ways to do that, one would be to create a basemap, and merge that in all branches:
(def BaseMap
  [:map
   [:db/type [:and string? [:fn {:error/message "should not be empty string"} not-empty-string?]]]
   [:h2/filename {:default "./boxdb"} string?]
   [:h2/user {:default "sa"} string?]])

(def Multi
  [:multi {:dispatch :db/shared-channel}
    ["folder" (mu/merge BaseMap [:map [:db/shared-channel [:= "folder"]] [:db/shared-channel-folder string?]])]
    ["web" (mu/merge BaseMap [:map [:db/shared-channel [:= "web"]] [:db/shared-channel-web string?]]])]])
#2020-03-0610:59mike_ananev@ikitommi 👍#2020-03-0611:00ikitommithe oldest non-resolved issue in malli: https://github.com/metosin/malli/issues/14#2020-03-0611:01ikitommikinda like select in spec2.#2020-03-0611:17teodorluhttps://malli.io seems to fail to infer the schema of values containing instants. Example value:
{:some-date #inst "2020-02-02"}
Is this expected behavior?
#2020-03-0611:22ikitommiI would not expect that. from a repl:
(require '[malli.provider :as mp])

(mp/provide [{:some-date #inst "2020-02-02"}])
; => [:map [:some-date inst?]]
#2020-03-0611:30Vincent Cantin@ikitommi for https://github.com/metosin/malli/issues/180, do we want to also add support for :+ :* :? and some other operators? in the :cat driven sequence?#2020-03-0611:31Vincent CantinI hope to start this issue if I have some time this weekend.#2020-03-0611:32ikitommithat would be awesome!#2020-03-0611:32Vincent Cantin(if I have time)#2020-03-0611:33ikitommidid an intial spike, using the net.cgrand/seqexp , something like:#2020-03-0611:33ikitommi
(def registry
  (merge
    default-registry
    {:cat (cat-schema :cat)
     :| (cat-schema :|)
     :* (regex-schema :* se/*)
     :*? (regex-schema : se/*?)
     :+ (regex-schema :+ se/+)
     :+? (regex-schema :+? se/+?)
     :? (regex-schema :? se/?)
     :?? (regex-schema :?? se/??)
     :repeat (regex-schema :repeat? se/repeat)
     :repeat? (regex-schema :repeat? se/repeat?)}))
#2020-03-0611:33ikitommididn’t get very far thou.#2020-03-0611:33Vincent CantinCould you push this to a branch ?#2020-03-0611:38ikitommiactually, that’s about it. here are the codes I was playing with:
(require '[net.cgrand.seqexp :as se])

(se/exec
  (se/cat
    (se/as :x int?)
    (se/as :y int?)
    (se/as :rest2 (se/* string?)))
  [1 2 "kikka" "kukka"])
; {:rest (), :match (1 2 "kikka" "kukka"), :x (1), :y (2), :rest2 ("kikka" "kukka")}

[:cat
 [:x int?]
 [:y int?]
 [:rest [:* string?]]]

(se/exec-tree
  (se/cat
    (se/*
      (se/as [:opts]
             (se/cat
               (se/as [:opts :x] string?)
               (se/| (se/as [:opts :s] string?) (se/as [:opts :b] boolean?))))))
  ["-server" "foo" "-verbose" true "-user" "joe"])

(se/exec
  (se/cat
    (se/as :x (se/repeat 1 int?))
    (se/as :y int?)
    (se/as :restz (se/* string?)))
  [1 2 "kikka" "kukka"])
; {:rest (), :match (1 2 "kikka" "kukka"), :x (1), :y (2)}

(se/*
  (se/as [:sections]
         (se/cat :h1
                 (se/as [:sections :ps]
                        (se/+ :p)))))
#2020-03-0611:39Vincent Cantinthank you#2020-03-0611:39ikitommijust updated https://github.com/cgrand/seqexp/issues/11, which is about using seqexp here#2020-03-0611:39ikitomminot going to do anything for this for now, so all yours 🙂#2020-03-0611:39Vincent CantinI will start by writing tests#2020-03-0611:39Vincent Cantin.. then the implementation to make them pass#2020-03-0611:44ikitommi👍#2020-03-0611:34ikitommisure, doesn’t do anything thou#2020-03-0611:39borkdudewhy not just "steal" the impl from spec?#2020-03-0611:39teodorluUsing malli.provider/provide does the trick! Thanks. I was surprised by what happened when I forgot to provide a sequence:
(mp/provide {:some-date #inst "2020-02-02"})
;; => [:vector some?]
com.github.metosin/malli {:git/url ""
                            :sha "9db4ff998b641d4a0cff4eb6d772fd0cb5d3b56c"}
#2020-03-0712:25ikitommiMaps turn into sequences in Clojure, so it’s a sequence of a vector of keyword? and inst? , which both are some?
(seq {:some-date #inst "2020-02-02"})
;; => ([:some-date #inst"2020-02-02T00:00:00.000-00:00"])
#2020-03-1011:55teodorluAh, that makes sense. Thanks.#2020-03-0611:41ikitommitaking the impl from spec is always an option.#2020-03-0611:42ikitommilast time I checked, the regexs where implemented using regular hashmaps in spec, would need at least some wrapping, don’t want to reserve maps for any one schema (family) type.#2020-03-0611:44ikitommigrand’s lib is 6y+ old, so “prior art” for this and all libs by Christophe have been great, also perf).#2020-03-0611:44borkdudeok#2020-03-0712:02ikitommiadded a contrinbution guide, 95% copy from reitit: https://github.com/metosin/malli/pull/186#2020-03-0812:06Vincent CantinRequest for comments on https://github.com/metosin/malli/pull/187#2020-03-0902:16Vincent Cantin@ikitommi https://github.com/cgrand/seqexp will need some adjustments if we want to use it in Malli: • Its implementation is not cross platform. • The maps returned by se/exec contain the keyed groups in the middle of :rest and :match , that could be a problem when the keys of those groups are controlled by Malli’s users.#2020-03-0906:16ikitommi@vincent.cantin ok. The cross-platform should be doable, ig Cristophe takes PRS (there should be an issue about that), not sure about the second. If that is not a good base lib to use, do you have plans on using something else? or from scratch?#2020-03-0906:33Vincent CantinI started to scratch my head about a new impl this morning. What's next depends on what features we want to offer the users.#2020-03-0906:35ikitommijust commented on the issue#2020-03-0906:37Vincent CantinI think that an optimized & naive impl might be faster on most use cases compared to a general algorith that works in theory faster and cover all edge cases. I would like to put them to a benchmark on realworld usages.#2020-03-0908:26ikitommiWhat kind of edge cases would not be covered using a naive impl? Benchmarks most welcome!#2020-03-1002:51Vincent CantinAll the use cases would be covered by using a naive impl. Some edge cases might have a high computational cost in a naive impl while a smarter impl would mitigate the cost of their parsing. The edge cases I am thinking about are consecutive sequences of variable length. But I am not sure if it is a real concern or my imagination.#2020-03-0906:48ikitommidid a quick draft of a swappable default registry: idea that the default registry could be bootstrapped into something that satisifes m/Registry protocol. There could be malli.mutable with an alternative impl using a extra atom to look for the schemas and malli.mutable/register!to register schemas into that. This would be a simple -> easy helper, already not happy with that. Comments welcome https://github.com/metosin/malli/pull/188#2020-03-0914:44ikitommi
;; JVM started with -Dmalli.core.registry=malli.mutable/registry

(require '[malli.core :as m])
(require '[malli.mutable :as mm])

(mm/register! ::age [:and int? [:> 18]])

(m/validate ::age 20)
; => true
#2020-03-0914:45ikitommi🙈#2020-03-1002:33Vincent CantinI also don’t like the idea to lose the property of working with immutable values.#2020-03-1002:39Vincent CantinI would suggest users who don’t want to have to pass the registry as option to Malli all the time to just create their own wrapper functions.
;; in my-ns
(defn validate [schema data options]
  (m/validate schema data (assoc options :registry my-registry)))
#2020-03-1002:42Vincent CantinI was facing the same situation in projects which use Vrac, and I used the wrapper pattern everywhere. It’s really not a big deal and it keep the satisfaction of working in an environment where we don’t even have to think about the side effects.#2020-03-0907:56eskosIs cross-platform regex even possible? Beyond basics I don’t think JS/Java/C# regex engines are even close to compatible with each other.#2020-03-0909:02Vincent Cantinwe are talking about validating data structures, not string content.#2020-03-0909:03eskos:thinking_face:#2020-03-0910:24ikitommi@U8SFC8HLP see https://clojure.org/guides/spec#_sequences#2020-03-0910:28eskosRight, thank you. I got confused, quite obviously 🙂#2020-03-0918:53shaunxcodeare recursive schemas possible w/ malli?#2020-03-0918:54shaunxcodee.g. defining a schema for a {:name "something" :kids [{:name "something" :kids [....]}]}#2020-03-0918:55shaunxcodeit does appear such things are possible in spec: https://gist.github.com/bhb/6cfcb3b38757442aec4ba5db46148699#2020-03-0919:08ikitommi@shaunxcode not yet, see https://github.com/metosin/malli/issues/20#2020-03-0919:11shaunxcodesorry just reading the whole of that linked issue and I see you guys are putting a lot of thought and effort into your approach so I will wait and see!#2020-03-0920:50ziltiI am about to start to try generating Malli schemas from PostgreSQL metadata. Very curious how far I can take this. If I find a way to store additional custom metadata in the database, I could end up generating 90% of a form simply by converting Postgres metadata, the rest being the layout.#2020-03-1108:13eskosJust curious, what kind of metadata you mean? PostgreSQL’s information schema is already quite extensive…#2020-03-1112:03bedersI have some code that does the same. I wasn't able to find a good solution for restricting string length. Maybe that has changed in the meantime.#2020-03-1113:06eskosIn PostgreSQL `TEXT` is string with infinite length (_=<https://wiki.postgresql.org/wiki/FAQ#What_is_the_maximum_size_for_a_row.2C_a_table.2C_and_a_database.3F|1GB> because that is any field’s maximum size_); string with specific maximum should be done with `VARCHAR` columns in which case `information_schema.columns` `character_maximum_length` has the column’s size limit information.#2020-03-1113:07eskosAssuming I’ve understood correctly, I seem to have a lousy track record on that part this week… 🙂#2020-03-1208:03viestiThe JDBC API also offers metadata, but probable less than direct PostgreSQL use. Prior art in spec land: • https://github.com/tatut/specqlhttps://github.com/viesti/table-spec#2020-03-1208:03viesti@U2APCNHCN the specql implementation might be really good help in looking what postgresql offers#2020-03-1208:58eskosJDBC https://docs.oracle.com/en/java/javase/11/docs/api/java.sql/java/sql/DatabaseMetaData.html is very extensive, haven’t tested but it seems with https://docs.oracle.com/en/java/javase/11/docs/api/java.sql/java/sql/DatabaseMetaData.html#getColumns(java.lang.String,java.lang.String,java.lang.String,java.lang.String) COLUMN_SIZE has the maximum length; the same TEXT/VARCHAR size detail as described above of course still applies.#2020-03-1216:09viestiyup, although I have a feeling that it might lack info on postgresql specific extensions#2020-03-1218:17ziltiYes, DatabaseMetaData is what I currently use. My showstopper right now is that I can't get the data transformations to work...#2020-03-1218:18ziltiI haven't heard of specql so far, but I will give it a look#2020-03-1307:39eskosMight be just me but if I were doing something very product specific, I’d steer far away from any kind of abstraction and just go as raw and bare in as possible… 🙂#2020-03-1310:25ziltiWell I think the problem wih going as raw and bare as possible is that you lose all the flexibility. And we're going to need that flexibility. My competition for creating forms are airtable and typeform. And I'd prefer we don't use either of those for our product#2020-03-1612:08teodorluQuickly model your tables with malli.provider/provide. Works well with JSON columns -- malli finds a specification that works with all your data.
(->> (with-open [conn (jdbc/get-connection db-datasource)]
         (jdbc.sql/query conn ["select * from myschema.mytable"]))
       (malli.provider/provide))
  [:map
   [:mytable/id int?]
   [:mytable/name string?]]
#2020-03-2010:36eskosGeneral consensus on whether this should work or not?
(require '[malli.util :as mu])
(mu/get-in [:map [keyword? int?]] [:a])
;=> returns nil, but I expected int?
I have nested maps with a few generic bits (my use case doesn’t allow composing schemas from reusable parts, yet) and noticed this behavior. Considering it’s supposed to mirror https://clojuredocs.org/clojure.core/get-in in functionality I’m hesitant to call this a bug, just wondering out loud… 🙂
#2020-03-2010:53ikitommi
(m/validate [:map [keyword? int?]] {:a 1})
; => false

(m/validate [:map [keyword? int?]] {keyword? 1})
; => true
#2020-03-2010:53ikitommido you mean :map-of here?#2020-03-2011:22eskosHm, I think I broke something else in the process as well… a moment 🙂#2020-03-2011:23eskosYes indeed, :map-of#2020-03-2011:25eskosThis of course changes the nature of the error, although I’m still not willing to call this a bug; I’m kind of hoping for magic which obviously isn’t there… :)
=> (mu/get-in [:map-of keyword? int?] [:a])
Execution error (IllegalArgumentException) at malli.core/fn$G (core.cljc:26).
No implementation of method: :-get of protocol: #'malli.core/LensSchema found for class: malli.core$_map_of_schema$reify$reify__692
#2020-03-2013:01ikitommiwould you like to do a PR of making :map-of implement the LensSchema protocol?#2020-03-2014:54eskosSo had a quick stab at it, -get is literally just
(-get [_ key default] (if (key-valid? key) value-schema default))
but -set is a lot harder as semantically it doesn’t really make sense as the result should probably be some kind of weirdo map schema, eg.
(mu/assoc [:map-of keyword? int?] :foo boolean?)
;=> should probably create
[:or
 [:map [:foo boolean?]]
 [:map-of keyword? int?]]
and that wouldn’t work for subsequent calls, assoc-in etc… One way to do this would be to allow :map to have default validators for kv pairs, effectively :map-of within :map, but that’s way too much work and decisions to do at this point of week 😅
#2020-03-2108:39eskosComing back to this, would it make sense to have it as an option for open maps that one could specify default k/v validators for :map schema? This way the -get could work if the given validator matches and -set would switch the default k/v validator to whatever is given if it doesn't match any of the actual keys.#2020-03-2013:42eskosHmm, I could take a look at it at least. Haven’t delved into malli code base that much yet so no promises on that side but what the hey, it’s not like I’d have any other places to be… 😄#2020-03-2013:44ikitommimaybe we should do a form-heavy work project together and finalize all the things 😉#2020-03-2107:25Vincent Cantin@ikitommi Clojure spec’s cat is doing 2 things at the same time and I don’t like that: • It says something about the container type of the sequence, • It says something about the content of the sequence. In Malli, I propose to move the description on the type of container to a separate structure schema name and to remove this responsibility from :cat and :alt . Maybe we can have something like:
[:cat [:a int?] [:b :string?]]              ;; matches '(1 "a") and [1 "a"]
[:in-list [:cat [:a int?] [:b :string?]]]   ;; matches '(1 "a")
[:in-vector [:cat [:a int?] [:b :string?]]] ;; matches '[1 "a"]
#2020-03-2605:57ikitommi@U8MJBRSR5 I guess you can say already [:and vector? [:cat ...]]#2020-03-2107:34Vincent Cantinor we can call them seq-of list-of and vector-of#2020-03-2107:44Vincent CantinI am preparing a commit in my MR, on which we can comment later.#2020-03-2205:59Vincent Cantin@ikitommi What is the reason Malli use [:map [:x int?] [:y int?]] instead of [:map :x int? :y int?] ?#2020-03-2206:00Vincent Cantin(I am asking for :map because I am wondering the same for :cat and :alt )#2020-03-2206:32mike_ananevI guess that inside [:x int?] you can set many things: customized error message, validation function etc. Without vector[:x int?] expression structure becomes unreadable.#2020-03-2206:55Vincent CantinGood point.#2020-03-2207:12Vincent CantinI updated the tests for :cat and :alt . Those tests represent the proposal I make for the data format which constitutes the API for this part of Malli. Feedback welcome ! https://github.com/metosin/malli/pull/187/files#2020-03-2414:53naomarikThere a way to use a custom :error/message for :or? Something like this:
[:or
 {:error/message "nil, string, or int"}
 nil?
 string?
 int?]
#2020-03-2606:01ikitommi@naomarik currently, no. And same applies for adding error messages on map entries. Would we solved with some way to figure out parents: https://github.com/metosin/malli/issues/120#2020-03-2606:01ikitommiideas welcome on how to do that.#2020-03-2606:03ikitommithought of copying props to childs from map entries and ors, but that would duplicate the info in m/form etc. knowing the parent at runtime would be cleanest way to resolve these.#2020-03-2712:56ikitommiin a project swapping plumatic into malli and I think there should be more built-in schemas for common thigns, strings and dates. Maybe something like:
[:string {:min 1, :max 10}]
[:int {:min 1, :max 10}]
[:date {:min "2012-04-23"}]
[:date-time {:min "2012-04-23 00:00:00", :format "yyyy-MM-dd HH:mm:ss"}] 
#2020-03-2713:10eskosSounds good to me. Maybe some of those should be aliases for other more in-depth schemas, but it’ll help the users a lot if there’s a good collection of those available out of the box.#2020-03-2713:19eskos…huh, even? and odd? are not in the predicate registry. I was about to mention how to something like [:range {:from 0 :to 10 :step 2}] would really be just [:and [:int {:min 0 :max 10}] [:even?]] and thought to check that. Sorry, my mind’s wandering 🙂#2020-03-2713:28eskosAlso pos? neg? seem to be missing. Are these left out on purpose or just forgotten?#2020-03-2714:15ikitommithe initial set of predicates for taken from clojure.spec, those which have generators defined. no other reason, could be added.#2020-03-2714:16eskosRighto 🙂#2020-03-3009:13teodorluI just had a pleasant experience during my team's standup. I'm working on code that depends on some PostgreSQL tables with JSON columns. We're migrating to a new set of tables given new needs. In that change, we're also moving some data from within the old JSON structure out into normal table columns. To give insight into my work, I generated schemas for the JSON with malli.provider/provide, and shared the generated malli schemas with my team. Explaining malli took 30 seconds and one example:
"Malli is a schema system for JSON."

"This JSON:"

{"x": 1,
 "name": "something"}

"Gets this schema:"

[:map
 [:x int?]
 [:name string?]]
This resulted in an experienced team member saying "oh, that somePropertyName, I didn't even know we had those." Felt great! 🎉
#2020-03-3013:23pithylessIf I want to add a custom tag to the registry, to support something like [:string {:min 1, :max 10}] - do I need to create a brand new IntoSchema (e.g. modeled similarly to m/fn-schema on -leaf-schema but with support for properties) + define a custom mg/-generator ? Or is there some way I can reuse more of the existing malli machinery?#2020-03-3017:55ikitommi@pithyless there are not many helpers in the core currently, but reifying the IntoSchema is a good way to do those. I think all the separate concerns (JSONSchema, Generators, Providers etc) should have a supporting protocol so one could easily satisfy all the concerns with one reify-block. Currently requires protocol + set of multimethods. For core stuff, it’s intentional to have the mm’s: the core remains small as it doesn’t have to know much/anythning about the other concerns.#2020-03-3017:56ikitommialso, there is a PR for discussion about swapping the default registry - to support spec-like mutable registries.#2020-03-3017:57ikitommiI think if there would be a set of type-kind-of schemas in malli core, one would not need the current core predicates at all. e.g. :string instead of string? , :date-time instead of inst etc.#2020-03-3110:00eskosI’ve been lately thinking whether malli should adopt git’s porcelain vs plumbing mentality for the schemas, but haven’t drafted anything sensible yet about it. To me there seems to be a lot of cases where the more complex/abstract schema really is just a composition of simpler ones, eg. range is just int with min and max and stepping validation (`[:range {:from 0 :to 10 :step 2}) => [:and [:int {:min 0 :max 10}] [:fn '(fn [i] (= 0 (mod i 2))]]`) or [:map-of keyword? string?] could be [:map {:default-kv [keyword? string?]}] &lt;- this one doesn’t exist! just an example) etc. but this would require quite a bit of thought on choosing what the base schemas in registries are and what counts as plumbing and what doesn’t… 🙂#2020-03-3113:46teodorluVega/Vega Lite does something similar. Vega Lite is high-level data format (for plotting), and compiles down to Vega. https://vega.github.io/vega-lite/#2020-04-0111:34mike_ananevWhat is an idiomatic way to put docstring into spec? via properties?#2020-04-0112:50teodorluI suspect that if you want to use Malli for your specs, you might want to model your documentation as pure data too. I'm eager to hear where other people would prefer to store those docstrings.#2020-04-0113:04ikitommi@mike1452 keys :title and :description are the default keys for docs, but free to use any other keys too.#2020-04-0113:04ikitommithose keys get pulled into JSON Schema & Swagger Schema docs.#2020-04-0113:04ikitommi(and should be used for other documentation too)#2020-04-0214:34ikitommifixed a bug on collection type transformations, which turned maps into vectors. Will push a new SNAPSHOT to clojars as need it with reitit (which uses leiningen)#2020-04-0214:34ikitommihttps://github.com/metosin/malli/pull/192#2020-04-0216:55dcjPresumably mailli schema properties are open? So I can specify a (new) key, that I use for my own purposes? Example: assign :postgres/type "some-postgres-type", later I walk schema and "make it so" via connection to database?#2020-04-0220:07dcjApparently so, a test schema for a database table:
(def metadata-schema
  [:map {:postgres/schema "slm"
         :postgres/table "measurement_metadatas"}
   [:id {:postgres/type :bigserial
         :postgres/key  :primary} int?]
   [:instrument-id {:postgres/type :bigint
                    :postgres/null? false} int?]
   [:datafile-id   {:postgres/type :bigint
                    :postgres/null? false} int?]
   [:seq           {:postgres/type :integer
                    :postgres/null? false} int?]
   [:origin        {:postgres/type :float8
                    :postgres/null? false} double?]
   [:scale         {:postgres/type :float4
                    :postgres/null? false} float?]
   [:starting-timestamp {:postgres/type :timestamptz
                         :postgres/null? false} inst?] ;; TODO: fix
   [:sample-rate   {:postgres/type :float4
                    :postgres/null? false} float?]
   [:log-interval  {:postgres/type :float4
                    :postgres/null? false} float?]
   [:weighting     {:postgres/type :text
                    :postgres/null? false} string?]
   [:manifest      {:postgres/type :int
                    :postgres/null? false} int?]
   [:time-zone     {:postgres/type :int
                    :postgres/null? false} int?]])
And, as a quick experiment, wrote fn to print DDL text driven by the above, yielding:
CREATE TABLE slm.measurement_metadatas (
    id bigserial PRIMARY KEY,
    instrument_id bigint NOT NULL,
    datafile_id bigint NOT NULL,
    seq integer NOT NULL,
    origin float8 NOT NULL,
    scale float4 NOT NULL,
    starting_timestamp timestamptz NOT NULL,
    sample_rate float4 NOT NULL,
    log_interval float4 NOT NULL,
    weighting text NOT NULL,
    manifest int NOT NULL,
    time_zone int NOT NULL
    );
#2020-04-0220:12dcjIs there (should there be) a long? predicate in malli?#2020-04-0300:37dcjOK, I have been yak shaving experimenting with Malli most of the day, having lots of fun! I have a bunch of questions: • After creating a new predicate, I want to assoc it into the existing predicate-registry map (creating my own). -register-var and -register are private. Obviously I can copy/paste these functions...suggestions? • Given my metadata-schema above, I want to transform a map via this schema. I get how m/decode is helpful, and I can specify key-transformer but I'm scratching my head about why/how I need to specify specific value transformers eg, string-transformer . Given that my schema has specified the type of each value (eg. int? ), then what I want to say is "coerce whatever you see into the specified type" I get that not all coercions can be pre-defined, and maybe there needs to be a way to add coercions....advice? • Again, given my metadata-schema above, let's say I want to obtain all the keys of that map schema. The top level form is a schema, and m/children gives me the list of children, but I can't figure out how to obtain the "name" of each child, without resorting to digging into the current implementation of that vector. In other words, sure I can first each child, but is there some better way to get that info/field? • Similar to the above, say I want to get each key in my map schema transformed in some way. When I want to transform a map, I do this: (m/decode my-schema my-map (mt/key-transformer {:decode csk/->snake_case_keyword}) because SQL column names are snake_case, but wouldn't it be better to include that transformer into my schema itself? Maybe I should define a new property at the map level :postgres/key-transformer and I (-> my-schema m/properties :postgres/key-transformer)? Thoughts? • Similar to the schema navigation/accessor questions above. when I first attempted to write the function to walk the schema and generate the DDL based on the :postgres/ properties, I tried to use m/accept with an function that would look at each [schema properties _] it encountered, but this didn't work because AFAICT the children are not schemas themselves. Finally I gave up and walked the schema using the obvious vector destructuring, but I don't feel this the right way to do it...#2020-04-0305:34ikitommi• properties are open, there is a bunch of reserved keys (`:title` , :description, :default, :min, :max etc.) and some namespaces that are reserved for extensions (`json-schema`, gen, swagger, encode, decode etc.). Should be documented and I guess a good convention could be “new unqualified might be used by malli in the future, to be safe, use qualified keys to avoid future clashes”. Goal is to describe the properties with malli schemas to get good error reporting on those too#2020-04-0305:37ikitommi-registerand -register-var could be made non-private, just undocumented. But in the end, it’s just assoc into the map. There is https://github.com/metosin/malli/pull/188 for discussion whether to support mutable registries oob#2020-04-0305:40ikitommi• coercions, sorry, didn’t get the question.#2020-04-0305:42ikitommi• all :map schemas implement MapSchema , which allows you to say (m/map-entries schema) to get the entries out#2020-04-0305:49ikitommi• if you want to transform just the entities, you could create a new Transformer, that only mount to :maps which have the :postgres/schema defined, this way, it doesn’t effect all maps. You can add a custom encoder & decoder just like with key-transformer. #2020-04-0305:56ikitommi• the m/accept… each IntoSchema is responsible for it’s own syntax so walking over schemas requires some knowledge of the schema in question. With maps, children is the sequence of entries, which are tuple3 of key-props-child.#2020-04-0306:00ikitommiYou can run (m/accept Schema m/map-syntax-visitor) to see the structure to be walked. With Java, the Visitor impl would have dispatch methods with different class signatures, with clojure, one can use multimethods for doing schema-based dispatch, see malli.generator or malli.json-schema for examples.#2020-04-0306:01ikitommiCurrently, only :map and :multi use custom syntax for it’s contents, later some variants of :or , :cat and :alt too (to support named branches). Could be others.#2020-04-0306:02ikitommiThere is also m/schema-visitor helper for walking.#2020-04-0306:03ikitommihope this helps @dcj#2020-04-0306:05ikitommithe long? predicate… could be, or :long type schema to companion :date-time, :date etc#2020-04-0306:05ikitommi(there is no long? in clojure core, for some reason)#2020-04-0306:07mike_ananev@ikitommi any plans to release the malli? malli is awesome! without release I clone malli to my projects every month manually.#2020-04-0306:21dcj@ikitommi Thank you for all the answers/advice/tips, I will try those tomorrow! I will also try to better explain my coercion question, but too tired to do it justice ATM...#2020-04-0307:46ikitommi@mike1452 not going to do that yet, but soon. with deps, you can depend on th the latest commit directly. With Leiningen, there is [metosin/malli "0.0.1-20200305.102752-13"], could start pushing new SNAPSHOTS after each merge.#2020-04-0307:48ikitommias soon as malli works fully with reitit, could push first alpha. some small hiccups still.#2020-04-0308:16eskosedit: all of this is garbage 🙂 see below `long?` predicate might be missing from Clojure due to the original design decision of having transparently coercing number types based on number size (#2020-04-0308:22eskosint? is true for Bytes, Shorts, Integers, Longs, while integer? is true for Integers, Longs, Clojure BigInts, BigIntegers, Shorts and Bytes. I don’t think there’s specifically benefit for checking if value is strictly Long, but YMMV.#2020-04-0309:42teodorluDifferentiating between Integer and Long would allow generating precise Java source code from a schema#2020-04-0317:29dcj@ikitommi m/map-entries and map-schema-entry structure as 3tuple of [key props child]is very helpful. For some reason, it makes me uneasy to just destructure the map-schema-entry 3tuple (instead of having accessor functions), but the good news is that you enforce that format, so even if the entry was specified without props, you provide the correct 3tuple, so destructuring works well! How would I test for a map schema? Is there a better way than: (= :map (m/name my-schema)) ?#2020-04-0317:34dcj@ikitommi (m/accept Schema m/map-syntax-visitor) is an awesome way to understand how the walking works, thanks! ATM, for my current "database table schema" task, I am pretty happy with m/map-entriesand processing the returned map-schema-entries.... But I'm sure I'll want to walk schemas at some point and good to know this.#2020-04-0317:47dcj@ikitommi I don't want to mutate the default registry, I just want to add a few more things to the default registry. My initial feeling is that I am happy to provide my registry as options map. Maybe there should be public functions to support users adding to the registry, to make their own variants I'll just copy/paste from -register and -register-var for now....#2020-04-0318:34dcj@ikitommi WRT my earlier coercion question(s), I studied the README and transform.cljc further, looks like this machinery will do everything I want, I'll just need to provide/add my own encoding and decoding functions...I'll get to that soon enough#2020-04-0318:47dcjRegarding long? and/or :long This requires further thought. If I am specifying a key :epoch-millisconds then it feels like that should specified as a long, not an int, so transformers can provide the correct type. Currently:
(def +string-decoders+
  (merge
    +json-decoders+
    {'integer? string->long
     'int? string->long
So, ATM those transforms do produce longs, but this seems somewhat arbitrary and not as precise as it might be. If the schema doesn't care about the the specific numeric type, then specifying something that includes a variety of types is fine.
#2020-04-0323:39dcjMore progress this afternoon: • Was able to add a new predicate to the default-registry, creating my-registry, and used it to define and validate a schema • Created a value transformer, and was able to get it invoked to transform a value via decode. Questions: • What is the meaning/use of the :name key in the mt/transformer input map? Can I put anything I want here? Should I use a namespaced keyword? • What other uses are there for the (optional) options map argument? (other than :registry)?#2020-04-0401:18dcjSo when I create a registry, and use it with m/schema, does that registry get saved with the schema (in -options)?#2020-04-0403:05dcjWRT:
What is the meaning/use of the :name key in the mt/transformer input map?
Is it the case that executing mt/transformer {:name :foo :decode ... :encode...} registers that transformer via the key :foo somewhere, and it can be retrieved via mt/transformer {:name :foo}) ?
#2020-04-0403:12dcjDoes a transformer that uses {:compile have to be defined inline within a specific schema, or can it be a "general" transformer that gets called against any schema ? The README example shows `
{:compile '(fn [schema _]
What is the second arg to that function (ignored here) ?
#2020-04-0406:31ikitommi:name in transformer is used for property-based transformation, I think only unqualified work right now (could be changed to support qualified too), having a transformer named :postgres allows one to define :encode/postgres and :decode/postgres keys in schema properties • currently no other used built-in keys than :registry in options, you can add you own - options are passed in mostly all user-defined callback functions. You should namespace your own option keys, just like custom schema property values, to avoid clashes • Schemas should save the options they were created with. I believe all in-built schemas do that. • :compile - it’s a feature of the interceptor, so can be used for general stuff, see the default-transformer (adding default values to schemas if missing): https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L315-L341 • second arg is the options#2020-04-0406:34ikitommiI would like to see to options be One place to customize how malli works: defining the registry, adding error messages & localizations (https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L5-L55), generarators etc, whether to use serializable functions or not (sci adds some to the bundle size in cljs) etc.#2020-04-0406:35ikitommihaving a malli schema to define the core malli options would also give good errors on invalid options 😉#2020-04-0406:37ikitommi@dcj here’s the line: only non-qualified transformer names are supported right now https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L60. Issue & PR welcome if you want qualified too.#2020-04-0406:40ikitommican you ask for the entries: (satisfies? malli.core/MapSchema schema)#2020-04-0406:41ikitommilooking at the source:
(defn map-entries
  "Returns a sequence of 3-element map-entry tuples of type `key ?properties schema`"
  ([?schema]
   (map-entries ?schema nil))
  ([?schema options]
   (if-let [schema (schema ?schema options)]
     (if (satisfies? MapSchema schema)
       (-map-entries schema)))))
#2020-04-0406:41ikitommiso, return nil or entries.#2020-04-0406:43dcj@ikitommi no current need for qualified transformer names. Thank you again for all these answers. I will need to experiment/study transformers more tomorrow... #2020-04-0406:57dcj@ikitommi Can :decode/foo keys in schema props be on the props for a single key in map, or only at the top :map level?#2020-04-0406:58ikitommiall schemas, but not currently the map-entries, later, see https://github.com/metosin/malli/issues/86#2020-04-0406:59ikitommibut this works:
[:map
 [:x [int? {:decode/postgre ...}]]]
#2020-04-0407:00ikitommithe parent-child thing (and support for recursive schemas) is the last big thing to resolve before a release. Ideas welcome on that.#2020-04-0412:44pithylessWhat kind of naming conventions have people found useful for malli-specs? In clojure.spec, one has a separate registry of names (s/def ::user) but malli just uses existing ns vars. Some idioms I've seen: user-schema, User, ns.just.specs/user ; a plain (def user) I'm afraid will clash too frequently with other/local bindings in your domain.#2020-04-0416:29dcj@pithyless Never having used malli 'in anger', my plan is to create a new/separate project/repo definitions to hold schemas and the like for a collection of interrelated projects. There will be subdirectories within definitions for specific domains, each with its own schema file/ns. So: definitions.foo.schema Within each schema file/ns I will just def the indvidual schemas, eg (def measurement (m/schema ...) I'll require these as needed like this: [definitions.foo.schema :as foo.schema] So then I'll access as foo.schema/measurement Disclaimer: Currently re-writing some code using my third schema tool, third database access library, and third time library, so clearly I can't get anything right the first N times 🙂#2020-04-0419:20dcjI'm having trouble groking property-based transformation. Here is a simplified example:
(def my-schema
  (m/schema
   [:map {:closed true
          :encode/postgres csk/->kebab-case-keyword
          :decode/postgres csk/->snake_case_keyword}
    [:id             {:optional true
                      :postgres/type :bigserial
                      :postgres/key  :primary} int?]
    [:instrument-id  {:postgres/type :bigint
                      :postgres/null? false} int?]
    ]
   ))

(def sample
  {:id 12345
   :instrument-id 67890})

(m/decode my-schema sample (mt/key-transformer {:name :postgres}))

;; {:id 12345, :instrument-id 67890}
I was expecting the :instrument-id key to be transformed to :instrument_id I have tried quoting the values of :encode/postgres and :decode/postgres various ways both single-quote and hash-single-quite to no avail. Also tried to use mt/transformer different problems. How should I do this?
#2020-04-0420:21dcjI guess my example above is confused about the encode/decode directions/terminology, be that as it may, I still can't seem to get the key-transformers to run, no matter what....#2020-04-0510:04ikitommi@dcj the key-transformer doesn’t understand the :name. It’s simple transformes all keys, not usefull in your case. You should use normal transformer instead. I created a gist, which might show the needed core for the thing you are looking for, using both named property-driven and a custom transformer options. Hope this helps: https://gist.github.com/ikitommi/5d71fc6b320e996b1ac9c17c4d412b71#2020-04-0510:13ikitommi@pithyless coming (and loving) Plumatic Schema, I have use the class-like naming: User, Address. Currently porting a Schema-project to use Malli, so the var names existed already. If I would add the schemas to the registry, might want to use qualified keywords: (assoc registry ::user User).#2020-04-0510:14ikitommiCurrently, haven’t added any domain schemas into the registry. is anyone doing that?#2020-04-0510:29kwrooijenI was thinking of doing this. I want to create UI elements based on the type of the schema. My initial thought was to use properties, but creating a registry could also be a valid option. But I'm still learning all the ins and outs of Malli though.#2020-04-0512:46kwrooijenIs there an idiomatic way of writing documentation for keys? I can't seem to find anything about that in the README. Currently I have this, which works, but feels very hacky.#2020-04-0513:17ikitommijust for keys:
(def Human
  [:map
   [:human/name {:description "Human name"} string?]
   [:human/age {:description "Human age"} int?]])

(doseq [[k p _] (m/map-entries Human)]
  (println k "->" (-> p :description)))
;:human/name -> Human name
;:human/age -> Human age
#2020-04-0513:17ikitommifor values:
(def Human
  [:map
   [:human/name [:and {:description "Human name"} string?]]
   [:human/age  [:and {:description "Human age"} int?]]])

(doseq [[k _ s] (m/map-entries Human)]
  (println k "->" (-> s m/properties :description)))
;:human/name -> Human name
;:human/age -> Human age
#2020-04-0513:18ikitommior:
(def Human
  [:map
   [:human/name [string? {:description "Human name"}]]
   [:human/age  [int? {:description "Human age"}]]])

(doseq [[k _ s] (m/map-entries Human)]
  (println k "->" (-> s m/properties :description)))
;:human/name -> Human name
;:human/age -> Human age
#2020-04-0513:22ikitommias Slack history loses all code, wrote a gist: https://gist.github.com/ikitommi/7c26aebf2b0f58d58432c884c8a55b5a#2020-04-0513:23kwrooijenAhh nice, m/map-entries is what I needed simple_smile also didn't know the :and was redundant. There are a couple of example that use :and + opts in the README#2020-04-0513:24kwrooijenSo would the convention be :description then? Since that was also used by json_schema ?#2020-04-0513:27ikitommiyes, :title and :description should be used, I’ll add those to README.#2020-04-0513:27ikitommi
(malli.json-schema/transform
  [:map
   [:human/name [string? {:description "Human name"}]]
   [:human/age [int? {:description "Human age"}]]])
;{:type "object",
; :properties #:human{:name {:type "string", :description "Human name"},
;                     :age {:type "integer", :format "int64", :description "Human age"}},
; :required [:human/name :human/age]}
#2020-04-0513:28kwrooijenOk, thank you for the clarification 👍#2020-04-0516:09sudakatuxHi#2020-04-0516:30sudakatuxSo how can i test malli on a lein project#2020-04-0516:30sudakatux?#2020-04-0516:35dcj[metosin/malli "0.0.1-SNAPSHOT"]#2020-04-0516:35sudakatuxcool just found it at the same time. Thank u#2020-04-0516:37dcj@ikitommi Thank you for the gist, that is super helpful! You provided two different examples, named property-driven transformations and named transformations, ATM I think I like the named transformations more, but will mull it over... The named transformations version of postgres-transformer answered further questions I had about how to use :complle Would it be a good/interesting idea to store transformers like these in the options of the schema? If so, then presumably they could be accessed later when needed for encode/decode calls....#2020-04-0600:16dcjRegarding map schemas: To get all the map keys from the schema itself: (mapv first (m/map-entries my-schema)) So far there have been two different cases where I need to get all the "encoded" keys from the schema itself, is there any way to do that? For the moment, I have defined the "key-encode-fn` and I store that away in the schema, then when I need the encoded keys, I go fetch it, and (mapv (comp key-encode-fn first) (m/map-entries my-schema) And when I am creating the encoders for a key related mt/transformer, again, I go get this function, and (partial -transform-keys key-encode-fn) This all works if I define and follow the convention above. Is there another/better way to get the "encoded" keys from a schema?#2020-04-0600:28dcjFor the record, defining transformers with :compile can be difficult to get correct, until you get an accurate mental model of when in time the various parts really happen on the 'def time' vs 'execution time' continuum. I struggled with this for a while, then eventually went back to a working example @ikitommi gave me, and put printlns at various places. That really helped! Finally I got it working! Very useful and powerful!#2020-04-0601:17dcjA map-entry is a 3tuple vector containing [`key ?properties schema]` , and as data/text, the properties is/are optional... If you've been following along with my "map-schema with support for Postgres tables" example(s), one map entry might like like this:
[:id {:optional true
        :postgres/type :column
        :postgres/datatype :bigserial
        :postgres/key :primary} int?]
Functionally, this is working great. From a schema readability perspective, IMHO, there is quite a lot of "textual distance" between the key, and the schema, when properties are specified/used, I really need to search for the schema of this key... So far, the schemas I have specified for keys are textually simple/short (mostly the built-in ones, or similar predicates I have created) Given that malli is "Pre-alpha, in design and prototyping phase" I wonder if switching the order of the propertes and the schema in a map-entry would improve readability/comprehension... Disclaimers: maybe this is a terrible idea. I have only a small number of hours of experience with malli, and only with pretty simple map schemas. I imagine this change would be extremely painful for everyone involved, and that could easily outweigh any benefits (assuming there are any benefits). I will happily live with the current map entry structure...
#2020-04-0607:35kwrooijenAn alternative is adding :id to the registry as {::id int?}. and then using the upcoming map syntax (if this pr gets approved https://github.com/metosin/malli/pull/194)
(def my-registry {::id int?})

(m/schema  
  [::id {:optional true
        :postgres/type :column
        :postgres/datatype :bigserial
        :postgres/key :primary}]
{:registry my-registry})
Something like this. And I think it will be easier to extend the registry later (if I'm reading the issues / PRs correctly)
#2020-04-0607:42kwrooijenAlso I think the reason for this format: [key ?properties schema] is because it's inline with the Hiccup syntax https://github.com/weavejester/hiccup Meaning that it would be a bit couter-intuitive to change the order. I could be wrong though.#2020-04-0615:27dcjSeems like it depends on how ‘terminal’ the schema element in a map-entry is. If the schema element was going to describe a another map/list/etc, then switching the props and schema would be horrible, which is probably why the current ordering is the way it is.... #2020-04-0616:01dcjWhich is the case with the example schema grounded_sage showed in a subsequent post. So now I see why the current ordering is best. Never mind....#2020-04-0616:04dcjNEVERMIND, I now see why current element ordering is best.... A map-entry is a 3tuple vector containing [`key ?properties schema]` , and as data/text, the properties is/are optional... If you've been following along with my "map-schema with support for Postgres tables" example(s), one map entry might like like this:
[:id {:optional true
        :postgres/type :column
        :postgres/datatype :bigserial
        :postgres/key :primary} int?]
Functionally, this is working great. From a schema readability perspective, IMHO, there is quite a lot of "textual distance" between the key, and the schema, when properties are specified/used, I really need to search for the schema of this key... So far, the schemas I have specified for keys are textually simple/short (mostly the built-in ones, or similar predicates I have created) Given that malli is "Pre-alpha, in design and prototyping phase" I wonder if switching the order of the propertes and the schema in a map-entry would improve readability/comprehension... Disclaimers: maybe this is a terrible idea. I have only a small number of hours of experience with malli, and only with pretty simple map schemas. I imagine this change would be extremely painful for everyone involved, and that could easily outweigh any benefits (assuming there are any benefits). I will happily live with the current map entry structure...
#2020-04-0615:54grounded_sageI’m having trouble updating a schema. Not sure how to do it. Current schema
[:map
   [:Data
    [:map
     [:CommunicationChannels
      [:vector
       [:map
        [:Identifier [:map [:Value string?] [:Details [:map [:ID string?] [:SystemID string?]]]]]
        [:ChannelType [:map [:Type int?] [:Name string?]]]
        [:Content string?]]]]]]]
Desired schema ->
[:map
   [:Data
    [:map
     [:CommunicationChannels
      [:vector
       [:map
        [:Identifier [:map [:Value string?] [:Details [:map [:ID string?] [:SystemID string?]]]]]
        [:ChannelType [:map [:Type int?]
                       [:or
                        [:Name "EMail"]
                        [:Name "Mobile"]
                        [:Name "Fax"]]]]
        [:Content string?]]]]]]]
#2020-04-0616:21grounded_sageOh figured it out. Have to use the :enum#2020-04-0620:40kwrooijenI'm trying to run the malli tests locally, but getting the following error: Exception: clojure.lang.ExceptionInfo: Kaocha ClojureScript client failed connecting back.#2020-04-0620:40kwrooijenThis doesn't happen in CI though. And I do have some tests that are actually failing, but this error is swallowing them all 😕#2020-04-0620:40kwrooijenDoes anyone else have this issue?#2020-04-0620:41kwrooijenMacOS Catalina, by the way#2020-04-0702:48grounded_sageI can’t get Union working on my schemas#2020-04-0702:54grounded_sageThis is the gist with the schemas in it.#2020-04-0702:56grounded_sagehttps://gist.github.com/groundedSAGE/4752ff810afe184ce119416a76eb8894#2020-04-0708:13eskos@grounded_sage it seems neither of those are valid schemas, I split the first one and reduced it into this which fails
user=> (m/validator [:map
  #_=>  [:Data
  #_=>   [:map
  #_=>    [:GroupMemberships [:vector [:or]]]]]])
Execution error (ExceptionInfo) at malli.core/fail! (core.cljc:73).
:malli.core/no-children
#2020-04-0708:14eskosThat :or seems lingering, typo?#2020-04-0708:15grounded_sageYea I thought that was strange. Can malli generate invalid schemas?#2020-04-0708:15grounded_sageBecause this was after doing schema inference.#2020-04-0708:17eskosIf that is a result of inferring then I guess the answer is yes 🙂 If you can figure out the minimal set of input data which produces invalid inferred schemas, I’m sure an issue is warmly welcomed.#2020-04-0709:39ikitommiSounds like a bug.#2020-04-0813:31grounded_sageJust posted an issue for it. https://github.com/metosin/malli/issues/196#2020-04-0712:55sudakatuxHi#2020-04-0721:22kwrooijenAnyone now if this is possible? :spritesheet/frame and :spritesheet/animation are technically both optional. But the map MUST have one or the other. This code snippet is invalid, the :or doesn't work (disclaimer, I'm working off of this PR https://github.com/metosin/malli/pull/194 which is why I can use this qualified-keyword syntax)#2020-04-0813:27aisamuThe way I've done such xor-like unions in regular spec is to move the or up one level:
[:or
 [:map
  :spritesheet/name
  :spritesheet/frame]
 [:map
  :spritesheet/name
  :spritesheet/animation]]
#2020-04-0723:16dcjLet's say I have a value that is an enum:
(def Weighting
  [:enum :dB-A :dB-C :dB-Z])
I can specify that as a value in another schema:
(def MyMap
  (m/schema
   [:map
    [:weighting Weighting]]))
Is there a way to turn the value schema into a "type" that malli understands? Let's say I need to transform the numbers 0,1,2 into the enumerated values...
(defn weighting->kw
  [n]
  (if (int? n)
    (case n
      0 :dB-C
      1 :dB-A
      2 :dB-Z
      n)
    n))
I specify a value transformer as:
(mt/transformer
 {:name :weighting
  :decoders {:enum weighting->kw}})
But in writing this, I am reaching into Weighting to pull out the :enum If I had two different enum values then dispatching via "value is an enum" doesn't seem like it will work, they will both resolve to enum. How does one handle this? Seems like I would want to specify the decoder as {:Weighting weighting->kw}
#2020-04-0813:27aisamuThe way I've done such xor-like unions in regular spec is to move the or up one level:
[:or
 [:map
  :spritesheet/name
  :spritesheet/frame]
 [:map
  :spritesheet/name
  :spritesheet/animation]]
#2020-04-0804:34ikitommiping @kevin.van.rooijen#2020-04-0804:36ikitommiI’m not a fond of too much extra syntax for :map. If the spec-like :or or :and are liked, I guess there could be :keys schema for that?#2020-04-0807:45kwrooijenThanks! This does exactly what I need. One question for clarification though; When talking about too much extra syntax, do you mean :or / :and or are you referring to my pull request?#2020-04-0807:46ikitommijust referring to the :or and :and , like spec1 has.#2020-04-0807:46ikitommiwill check the PRs as soon as have time. busy times.#2020-04-0807:49kwrooijenAhh ok thanks. Take your time!#2020-04-0812:32yonatanelI need something similar and even a bit more complex. I want to require a value either at top level or nested:
;Valid:
{:key1 "yo"}

;Also valid
{:nested {:key2 "yo"}}
It’s not the same key but they have the same meaning and one must exist. Currently I have this schema:
[:or
 [:map
  [:key1 any?]
  [:nested {:optional true} [:map [:key2 {:optional true} any?]]]]
 [:map
  [:key1 {:optional true} any?]
  [:nested [:map [:key2 any?]]]]]
but I have more keys like this, and some options can be either of 3 keys so I need 3 maps for each key and then combine all of it with :and.
#2020-04-0804:42ikitommi@dcj easiest way to attach a decoder just for that enum would be:
(def Weighting
  [:enum {:decode/enum weighting->kw} :dB-A :dB-C :dB-Z])
and then applying it with:
(m/decode MyMap {:weighting 0} (mt/transformer {:name :enum}))
; => {:weighting :dB-C}
#2020-04-0804:45ikitommiif you want to have a transformer for that, you can do the same as I gisted about the :postgres/table => add some hint to the enum and create a transformer that attaches in :compile just for that schema. hope this helps.#2020-04-0813:32kwrooijenYeah that's a good option as well. But that will probably lead to a lot of duplication (since my example was just a stripped down version)#2020-04-0814:18aisamuYup, it was quite annoying#2020-04-1010:19yonatanel@kevin.van.rooijen meanwhile I have this require-either function that adds :fn schema to the map with a humanized error message and makes the relevant keys optional (https://gist.github.com/yonatane/7bd1d38449f56b3aea62eb391964aee4)#2020-04-1119:32dcjJust to confirm: When specifying keys in a map schema, it is fine to http://use.so a namespaced keyword, eg: [:instrument/id int?] AFAICT from one quick test, this "just works". The schema can be defined, and validate works as expected... Correct?#2020-04-1120:22pithylessYes. There's even an open documentation issue for this - https://github.com/metosin/malli/issues/189#issue-590324344#2020-04-1220:02sudakatuxHi#2020-04-1220:03sudakatuxSo im testing malli with cohersion and swagger. I was wondering when following the example online. the type dispatcher for swagger. its an empty object. how does one pass the descriminator value#2020-04-1220:04sudakatuxSo multi here https://malli.io/?value=%5B%7B%3Atype%20%3Asized%2C%20%3Asize%2010%7D%0A%20%7B%3Atype%20%3Ahuman%2C%20%3Aname%20%22tiina%22%2C%20%3Aaddress%20%7B%3Astreet%20%22kikka%22%7D%7D%5D&amp;schema=%5B%3Avector%0A%20%5B%3Amulti%0A%20%20%7B%3Adispatch%20%3Atype%7D%0A%20%20%5B%3Asized%20%5B%3Amap%20%5B%3Atype%20%5B%3A%3D%20%3Asized%5D%5D%20%5B%3Asize%20int%3F%5D%5D%5D%0A%20%20%5B%3Ahuman%0A%20%20%20%5B%3Amap%0A%20%20%20%20%5B%3Atype%20%5B%3A%3D%20%3Ahuman%5D%5D%0A%20%20%20%20%5B%3Aname%20string%3F%5D%0A%20%20%20%20%5B%3Aaddress%20%5B%3Amap%20%5B%3Astreet%20string%3F%5D%5D%5D%5D%5D%5D%5D#2020-04-1220:04sudakatuxthe type attribute. say i want to pass that value. how should i pass it#2020-04-1220:05sudakatuxbecause in swagger its shown like :properties {:type {},#2020-04-1307:37ikitommimaybe :swagger/discriminator :type?#2020-04-1307:38ikitommihaven’t used the discriminator, so not sure how it should work. what is the expected swagger schema?#2020-04-1310:45sudakatuxso i would expect i could "type":"mytype"#2020-04-1310:45sudakatuxGM#2020-04-1310:50sudakatuxSo my probelm was that i could not POST since it could not figure out the type#2020-04-1310:50sudakatuxthe way i got what i wanted was by doing this#2020-04-1310:51sudakatux
(m/validate schema (m/decode
    schema data
    (mt/transformer mt/strip-extra-keys-transformer mt/string-transformer)))))
#2020-04-1310:51sudakatuxwhere schema is#2020-04-1310:51sudakatux
[:multi
 {:dispatch :type, :decode/string (fn* [p1__40150#] (update p1__40150# :type keyword))}
 [:other [:map [:data [:map [:nombre string?] [:apellido string?] [:edad int?]]] [:type [:= :other]]]]
 [:one [:map [:data [:map [:nombre string?] [:apellido string?] [:kk int?]]] [:type [:= :one]]]]]
#2020-04-1310:52sudakatuxand data#2020-04-1310:53sudakatux
{:data {:nombre "nombre" :apellido "apellido" :edad 12} :type "other"}
#2020-04-1310:54sudakatuxon an ideal I would have liked to be able to use that schema in the {:paremeters {:body here}}#2020-04-1313:01plexusam I right that this is not supported? should it be?
(def m [:map
          [:i {:error/message "not a number"} int?]])

  (me/humanize (m/explain m {:i "123"}))
  ;; => {:i ["should be int"]}
#2020-04-1313:02plexusI find in general there's some annoying asymmetry between properties on map entries vs properties schemas. See also my PR for adding :gen/gen / :gen/elements / :gen/fmap support to map entries#2020-04-1313:07sudakatuxsorry deleted the message by mistake#2020-04-1313:07sudakatuxwill take a look#2020-04-1313:08plexusthanks @U8JTE9PG8. I'm interested in general if users should expect props on schemas to also work on map entries, or if that's not a design goal.#2020-04-1516:26ikitommiThanks @U07FP7QJ0 for the PR! Will check it soon along others. I would like to have first class entry meta, not sure what is the best way for it.#2020-04-1313:05plexusI suppose you can do this
(def m [:map
          [:i
           [int? {:error/message "not a number"}]]])
#2020-04-1513:44orestisI saw the ClojureD talk where it’s mention that there was some experiments tying malli and clj-kondo together. Is that code somewhere available? I’d love to have compile-time checks of some map usage. I don’t care particularly about the syntax right now, mostly about the hooking up kondo and its analyzer to validate malli schemas.#2020-04-1516:22ikitommi@orestis it was a quick hack, not published to repo. The m/defn is 1:1 from plumatic schema, emitting a clj-kondo file into the project. No generic conversion and plumatic port just a poc. Would take a day to make it real.#2020-04-1516:28orestisSo essentially it was just emitting clj-kondo specific schemas for the function, right? As if you hand-typed the kondo conf manually. I can work with that...#2020-04-1516:32ikitommiyes. Generic malli Schema -> clj-kondo schema transformer would be great - and relatively easy to do, just like the json-schema transformer.#2020-04-1516:29ikitommi@jstuartmilne if you know the form of the swagger schema, you could add the swagger transformation for :multi. It's an multimethod doing that, so can be done in user space. PR welcome when it works ;)#2020-04-1516:30ikitommiAlso, you can write the swagger manually, just add a :swagger key to schema and a full schema as value. Overrides everything#2020-04-1622:51sudakatuxcool thanks. it working how i want now 🙂#2020-04-1609:41grounded_sageThis bug https://github.com/metosin/malli/issues/196 is pretty critical for us atm. I’m happy to try jump in and fix if someone can point me in the right direction with some ideas on how to fix it.#2020-04-1612:02ikitommi@grounded_sage I think the culprit is here: https://github.com/metosin/malli/blob/master/src/malli/provider.cljc#L41#2020-04-1612:03ikitommifor empty sequences, it tries to figure out what is a valid schema and finds empty :or#2020-04-1612:05ikitommi
(require '[malli.provider])
(malli.provider/provide [[]])
; => [:vector [:or]]
#2020-04-1612:25ikitommi@grounded_sage should be fixed in master.#2020-04-1614:22grounded_sageLegend! #2020-04-1612:27ikitommi
(provide [])
; => any?

(provide [[]])
; => [:vector any?]

(provide [[[[[[[]]]]]]])
; => [:vector [:vector [:vector [:vector [:vector [:vector any?]]]]]]
#2020-04-1614:25grounded_sageThis is so helpful for the work I am doing atm where we are ingesting messy JSON from third parties. I use Malli to show me all the variations of schemas their JSON can have. Check the frequencies to see which ones we want to support. Can deep diff the schemas to see the differences. Then use the generated unified schema as the base for robust Meander pattern matching. #2020-04-1616:31ikitommiSounds cool! do you write the meander definitions manually or programmatically from schemas?#2020-04-1616:40grounded_sageCreate them manually. Would be interesting to see if I can generate a nice starting point from a Malli schema.. haven’t dove in that much yet to see if it’s practical.#2020-04-1616:43grounded_sageReminds me I did find another bug or unexpected behaviour earlier. Where doing unions over a number of schemas resulted in created some nested schema definitions that weren’t there. I got around this by just capturing the original JSON for the various schemas#2020-04-1920:39teodorluAre anyone using Malli for recursive schemas? Attached schema generated with malli.provider/provide. In spec, I'd do something like this (not tested, and missing some properties):
(s/def ::title string?)
(s/def ::children (s/coll-of ::node))

(s/def ::node
  (s/keys :req-un [::title
                   ::children]))
#2020-04-1921:30roklenarcicI am trying to add an error message to an existing schema, I tried simply saying: [:and {:error/message "X"} myschema] and that didn’t work#2020-04-2006:10eskos:and requires two leaves, you have only one.#2020-04-2006:28eskos@ikitommi What would be your preferred media for discussing some design choices malli could potentially have a stake in? I’ve been thinking for quite a while that there’s two topics I’d like to have persistently discussed, decided and documented, namely the plumbing&porcelain concept for schemas themselves I’ve mentioned before and swagger2 vs openapi3 vs postman collections generation, and while we could discuss those things here, the ephemeral nature of Slack and this unpaid Clojurians Slack instance especially isn’t very good documentation wise… 🙂#2020-04-2014:10aisamuI'll just chime in to mention that some folks here (#data-science IIRC) have been using Zulip for that with apparent success! Doesn't lose history and has a much better (i.e. usable) threading model#2020-04-2015:22teodorluSecond chime in: I've been using Zulip quite a bit, and I think I prefer it to Slack. I don't see a huge difference, but it would be possible to have a single malli channel with different topics, where design could be one.#2020-04-2015:25aisamuYup! I actually consume this slack's content through Zulip's mirror.#2020-04-2015:25teodorluHuh, cool. Didn't know that was possible. Do you happen to have a link?#2020-04-2015:30aisamuSure! https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/malli Each one of slack-archive's topics is a channel (after you invite @UFNE73UF4, that is)
#2020-04-2015:41eskosLooks like Zulip’s free tier has limited search, are you sure it’s not going to lose history?#2020-04-2015:41eskosOr is someone paying/hosting that? 🙂 Sorry, not familiar with Zulip’s hosting scheme…#2020-04-2015:44aisamu> Zulip Cloud Standard is free for open source projects! 🎉#2020-04-2106:11eskos@ikitommi that could maybe actually work for Metosin’s OSS? --^#2020-04-2110:30eval2020Zulip-admin here. We're indeed using Zulip's OSS-plan (https://www.zulipchat.com/for/open-source/) Happy to answer any more questions.#2020-04-2110:31eval2020btw I see @UFNE73UF4 is no longer a member of this channel... :thinking_face:#2020-04-2110:58eval2020scratch that (bots don’t appear in the member-list)#2020-04-2114:03teodorluThanks for keeping Zulip working 🙂 It's been really helpful for doing Scicloj-work 💯#2020-04-2114:38eval2020yw!, great to hear.#2020-04-2208:49plexuslate to the thread here but I'd like to propose ClojureVerse for this. Long form asynchronous is more suitable for writing out proposals and discussing their merit. Happy to set up a Malli category.#2020-04-2212:21aisamuWorth noting that Zulip also favours long form, async content (with proper quoting, nesting, etc) given the threading model. (And I also consume ClojureVerse through Zulip's mirror - annoucements, but still) simple_smile#2020-04-2310:05ikitommioh, what would the category mean? all the discussion still go to the main feed?#2020-04-2310:08ikitommiBut, as the discussions are anyway in threads in clojureverse, one could just not read the malli thread/s if that’s not relevant to them.#2020-04-2311:03teodorlu@ikitommi Here are all the project threads: https://clojureverse.org/c/projects/24 And here's the shadow-CLJS category: https://clojureverse.org/c/projects/shadow-cljs/20#2020-04-2006:41ikitommi@teodorlu recursion schemas, discussion here: https://github.com/metosin/malli/issues/20#2020-04-2007:27teodorluThanks for the reference!#2020-04-2006:48ikitommi@roklenarcic @suomi.esko there are two ways that the properties are used: 1. top-down: generators use this. the first generator that is found is used. [:and {:gen/elements ["a" "b" "c"], :error/message "fail"} string?] short-circuits on that gen. decoders & encoders compose top-down, json-schema overrides work this way etc. 2. after-the-fact: error messages are looked for after the error has happened. here: it the example, the failure happens at string? , which doesn’t know any of the parent decorations for errors. this is bad, and should be fixed in a coherent way, related issues: https://github.com/metosin/malli/issues/120 & https://github.com/metosin/malli/issues/86#2020-04-2006:50ikitommithe error message thing could be “easily” fixed so that the error message formatter looks also at the parent schemas (all the information is available in malli.core/explain result.#2020-04-2006:51ikitommi… but would not solve [:map [:x {:error/message "fail"} string?]]], where the info is in the entry meta.#2020-04-2006:51ikitommiideas welcome, will put my hammock on the backyard this week, so will think about how to do this#2020-04-2006:56ikitommi@suomi.esko maybe a open malli video chat? google hangouts for example later this week? issues have persistent discussion, but as things link, the discussion is bit scattered.#2020-04-2007:13eskosHmm, video chat could probably be an interesting way to solve this, as I have no idea if this is like 5, 15 or 50 minute topic and it’s not quite clear to myself either entirely what the whole scope is…#2020-04-2014:10aisamuI'll just chime in to mention that some folks here (#data-science IIRC) have been using Zulip for that with apparent success! Doesn't lose history and has a much better (i.e. usable) threading model#2020-04-2020:51ikitommi
(def Country
  [:map {:id "Country"}
   [:name [:enum :FI :PO]]
   [:neighbors any?]])

(def Burger
  [:map {:id "Burger"}
   [:name string?]
   [:description {:optional true} string?]
   [:origin [:maybe [:maybe Country]]]
   [:price pos-int?]])

(def OrderLine
  [:map {:id "OrderLine"}
   [:burger Burger]
   [:amount int?]])

(def Order
  [:map {:id "Order"}
   [:lines [:vector OrderLine]]
   [:delivery [:map
               [:delivered boolean?]
               [:address [:map
                          [:street string?]
                          [:zip int?]
                          [:country Country]]]]]])
#2020-04-2020:52ikitommi
(require '[malli.mermaid :as mermaid])

(mermaid/class-diagram Order)
;"classDiagram
;   class Country {
;     + :name [:enum :FI :PO]
;     + :neighbors any?
;   }
;   class Burger {
;     + :name string?
;     + :description string?
;     + :origin Country
;     + :price pos-int?
;   }
;   Burger o-- Country
;   class OrderLine {
;     + :burger Burger
;     + :amount int?
;   }
;   OrderLine o-- Burger
;   class OrderDeliveryAddress {
;     <<embedded>>
;     + :street string?
;     + :zip int?
;     + :country Country
;   }
;   OrderDeliveryAddress o-- Country
;   class OrderDelivery {
;     <<embedded>>
;     + :delivered boolean?
;     + :address OrderDeliveryAddress
;   }
;   OrderDelivery *-- OrderDeliveryAddress
;   class Order {
;     + :lines OrderLine
;     + :delivery OrderDelivery
;   }
;   Order o-- OrderLine
;   Order *-- OrderDelivery"
#2020-04-2020:53ikitommi#2020-04-2020:53ikitommihttps://github.com/metosin/malli/pull/198#2020-04-2020:54ikitommi(59 loc 😉)#2020-04-2022:00kwrooijenWow that's awesome#2020-04-2022:04kwrooijenThe tree structure logic is the mermaid namespace, right? I think it could be useful to have generic function to generate a dependency tree#2020-04-2208:49plexuslate to the thread here but I'd like to propose ClojureVerse for this. Long form asynchronous is more suitable for writing out proposals and discussing their merit. Happy to set up a Malli category.#2020-04-2212:21aisamuWorth noting that Zulip also favours long form, async content (with proper quoting, nesting, etc) given the threading model. (And I also consume ClojureVerse through Zulip's mirror - annoucements, but still) simple_smile#2020-04-2105:18ikitomminow in malli.mermaid, but could push some of the building blocks into malli.util: • function to resolve names for the anonymous nested schemas • function to pull out relations between schemas that match a precidate#2020-04-2105:20ikitommithe actual mermaid code would be ~20 lines. would be easy to add a DOT printer too.#2020-04-2105:21ikitommimermaid doesn’t seem to allow special symbols in the class names, would have liked Order$Delivery$Address instead for the derived class names, easier not to clash. DOT allows that.#2020-04-2121:00geraldodevHi , I was trying to create a malli zipper and was using children to get down the tree. It happens that it crashes when there is no children. Is there a function for that ?#2020-04-2121:03geraldodevhttps://gist.github.com/geraldodev/f637da76bdac78fe8f7e330db43fe311 A malli zipper if anyone want to criticize is more than welcome#2020-04-2523:03sudakatuxHi#2020-04-2523:04sudakatuxi have a question regarding the json-schema suport#2020-04-2523:04sudakatuxSo if i go to the example:#2020-04-2523:04sudakatux[:map {:title "Address"} [:id string?] [:tags [:set keyword?]] [:address [:map {:json-schema/title "Something"} [:street {:json-schema/title "Something Else"} string?] [:city string?] [:zip int?] [:lonlat [:tuple double? double?]]]]]#2020-04-2523:09sudakatuxand add title to :street that will not work i would expect to see "Something Else" as a title property for street#2020-04-2523:09sudakatuxI do see "Something" as the title for Addess#2020-04-2523:09sudakatuxGenerated JSON-schema :#2020-04-2523:10sudakatux
{:type "object",
 :properties {:id {:type "string"},
              :tags {:type "array",
                     :items {:type "string"},
                     :uniqueItems true},
              :address {:type "object",
                        :properties {:street {:type "string"},
                                     :city {:type "string"},
                                     :zip {:type "integer",
                                           :format "int64"},
                                     :lonlat {:type "array",
                                              :items [{:type "number"}
                                                      {:type "number"}],
                                              :additionalItems false}},
                        :required [:street :city :zip :lonlat],
                        :title "Something"}},
 :required [:id :tags :address],
 :title "Address"}
#2020-04-2523:11sudakatuxWhy is not "Something Else" a title for street am i missing something?#2020-04-2815:23ikitommi@jstuartmilne it’s a known issue, to be solved, see https://github.com/metosin/malli/issues/86#2020-04-2815:23ikitommifor now, this works: [:street [string? {:json-schema/title "Something Else"}]]#2020-04-2815:24ikitommie.g. put the meta into entry schema, not to entry itself.#2020-04-2815:24ikitommithis needs a day in the hammock to figure out how to best resolve that.#2020-04-2815:25ikitommi(good thing I just found my hammock from the basement :)), also ideas & help welcome.#2020-04-2815:30sudakatuxahhh cool#2020-04-2815:30sudakatuxwill try that#2020-04-2815:30sudakatuxthank u verry much#2020-04-2915:47daveliepmannHow much control does Malli currently give over map keys? For instance that they should be of a particular set, or within a range, or keywords or strings, but without specifying the key precisely? I have an existing non-relational database for which I'd like to infer schemas with mp/provide , but it seems to get confused around hashmaps that are built to store user input (e.g. a mapping from username to info).#2020-04-2916:18ikitommi@daveliepmann there is :map-of , like in spec.#2020-04-2916:20ikitommithe mp/provide is not smart enough yet to figure those out. Could be..#2020-04-2916:23daveliepmannThanks!#2020-04-2920:27teodorlu@daveliepmann @ikitommi I was also getting more corarse-grained results with provide than I would like. I would love to be able to provide a "starting point" for the schema. Please excuse my backseat driving:#2020-04-3008:36ikitommi@teodorlu Agree, would be useful. I think someone could make a thesis (and read other people theses ;)) out of the providers. The current impl is really naive (69 loc), help most welcome. Ambrose might have some math for this already on typed clojure#2020-04-3008:36ikitommi#2020-04-3008:36ikitommi#2020-04-3008:38teodorluI'll have a look at the provider source 🙂#2020-04-3014:39daveliepmann@teodorlu I had a similar thought under which your hypothetical looked like [:map [:strength int?] #{"mage" "paladin" "ranger"}] in my head 🙂#2020-04-3014:46daveliepmannThe shortest real example I have is [:map-of #{<domain-specific items>} [:map-of #{<more domain-specific items>} boolean?]] . (The first set of domain items is limited but long enough to make me not want to enumerate each key to the same spec of values; the second set of domain items is IDs from another system that can't be enumerated in a spec but that I might want to check the existence of.) On some of the larger schemas I can see some difficulty getting the API for mp/provide to play nicely with a skeleton schema, because of how nested it would have to be.#2020-05-0123:49geraldodev(m/decode int? "23.abc" mt/string-transformer) gives me a string on clojure on cljs it gives me an 23 integer. it strips the integer part and passes. is it normal ?#2020-05-0411:47katoxHi all, I neeed a simple clojure.test check whether a struct conforms to Malli schema. I came up with this helper:#2020-05-0411:47katoxhttps://gist.github.com/katox/d5817209aec78edb052f851c86e34f99#2020-05-2412:42rutledgepaulvfyi I'm working on something similar to https://github.com/rjsf-team/react-jsonschema-form for malli using semantic-ui-react components w/ reagent and it's coming together nicely so far. good job tommi and other contributors 🙂 . i'll share something as i make more progress (just started yesterday)#2020-05-2508:29ikitommilooking forward to this!#2020-05-2508:42ikitommicould someone check the reasoning in https://github.com/metosin/malli/pull/201?#2020-05-2508:45ikitommiI now understand why schema.core/both got deprecated. :or is kinda complex when you wan’t to cover both validation and transformation. Same thing was “fixed” in spec-tools some weeks ago: results of a transformed branch needs to be validated to know if the branch was good.#2020-05-2811:12Vincent CantinI faced the same issue in my lib, and tried to separate shape-type elements from the predicates which are checking properties on them.#2020-05-2508:46ikitommischema had s/conditional, maybe Malli should have :conditional too :thinking_face:#2020-05-2508:48ikitommicould be implemented using :multi with just few extra lines to swap the dispatch function & values.#2020-05-2614:54sudakatuxHi#2020-05-2615:03sudakatuxim looking for a way to filter out#2020-05-2615:04sudakatuxrecursively in a nested schema every schema containing a certian property#2020-05-2615:04sudakatux`
[:identity [:map [:user map?] [:profile map?] [:token  [string? {:deleteMe true}] ]]]]
#2020-05-2615:05sudakatux(my-cool-filter [:identity [:map [:user map?] [:profile map?] [:token [string? {:deleteMe true}] ]]]])#2020-05-2615:05sudakatux
[:identity [:map [:user map?] [:profile map?]]]]
#2020-05-2615:06sudakatuxI suppose i should be looking at the visitor. since i can see it runns through every node. but im not sure how to filter it out#2020-05-2805:26ikitommi@jstuartmilne something like:
(m/accept
  [:map
   [:user map?]
   [:profile map?]
   [:nested [:map [:x [:tuple {:deleteMe true} string? string?]]]]
   [:token [string? {:deleteMe true}]]]
  (fn [schema children _ options]
    (if-not (:deleteMe (m/properties schema))
      (let [children (filterv (if (m/map-entries schema) last identity) children)]
        (m/into-schema (m/name schema) (m/properties schema) children options)))))
; => [:map [:user map?] [:profile map?] [:nested :map]]
#2020-05-2805:27ikitommie.g. check if the schema has the property, remove if it has. if not, you have to remove the nil childs as it’s currently not valid syntax. There are two child syntaxes: the normal and the (map)entry-one. need to handled differently.#2020-05-2805:28ikitommithat doesn’t drop the empty nodes, e.g. :map without any keys is valid, same with empty :tuple etc.#2020-05-2810:09ikitommianyone interested in doing malli.xml? something like:
(def User [:map [:users [:vector [:map [:user string?]]]]])

(m/decode User "<users><user>simo</user></users>" malli.xml/xml-transformer)
; => {:users [{:user "simo"}]}

(m/encode User {:users [{:user "simo"}]} malli.xml/xml-transformer)
; => "<users><user>simo</user></users>"
#2020-05-2810:09ikitommirelated: https://github.com/metosin/muuntaja/pull/114#2020-05-2812:06sudakatuxThank u so much.Ive been struggling witth that#2020-05-3013:07ikitommiConverting between vector (hiccup) and map-syntax:
(defn ->map-syntax
  ([?schema] (->map-syntax ?schema nil))
  ([?schema options] (m/accept ?schema m/map-syntax-visitor options)))

(defn <-map-syntax
  ([m] (<-map-syntax m nil))
  ([{:keys [name properties children]} options]
   (let [<-child (if (-> children first vector?) (fn [f] #(update % 2 f)) identity)]
     (m/into-schema name properties (mapv (<-child #(<-map-syntax % options)) children)))))
=>
(def Schema
  [:map
   [:id string?]
   [:tags [:set keyword?]]
   [:address
    [:vector
     [:map
      [:street string?]
      [:lonlat [:tuple double? double?]]]]]])

(->map-syntax Schema)
;{:name :map,
; :children [[:id nil {:name string?}]
;            [:tags nil {:name :set
;                        :children [{:name keyword?}]}]
;            [:address nil {:name :vector,
;                           :children [{:name :map,
;                                       :children [[:street nil {:name string?}]
;                                                  [:lonlat nil {:name :tuple
;                                                                :children [{:name double?} {:name double?}]}]]}]}]]}

(-> Schema
    (->map-syntax)
    (<-map-syntax))
;[:map
; [:id string?]
; [:tags [:set keyword?]]
; [:address
;  [:vector
;   [:map
;    [:street string?]
;    [:lonlat [:tuple double? double?]]]]]]
#2020-05-3013:18ikitommiShould the support for map-syntax be baked into malli.coredirectly? One could create a schema with m/schema using either one (can’t mix - the whole Schema tree needs to be created with same syntax - not to lose any extensibility) and the created Schema would know the syntax it’s created with: m/form would return in the original one.#2020-05-3013:24ikitommihmm. maybe not now (thinking aloud), have some really big map schemas in a project with a lot of properties everywhere, that could benefit from map-syntax, but should verify it first.#2020-05-3013:25ikitommiwill just push the helpers into malli.#2020-05-3013:43ikitommihttps://github.com/metosin/malli/pull/202#2020-05-3015:29ikitommimerged. Also, added a docs/tips.md to collect stuff from here. All tips, docs welcome! https://github.com/metosin/malli/blob/master/docs/tips.md#2020-05-3015:30ikitommiAsciidoc might be a better format as it has the table of contents etc.#2020-05-3103:34naomarikIt took a little while to figure this out by examining malli source, is this how you guys are passing in outside vars to the error/fn property?
(register! :phone [:re
                   {:description "Phone",
                    :phone-prefix-regex phone-prefix-regex
                    :phone-code-regex phone-code-regex
                    :error/fn
                    '(fn [{:keys [value schema]} _]
                       (let [{:keys [phone-prefix-regex
                                     phone-code-regex]} (m/properties schema)
                             has-prefix? (re-find phone-prefix-regex value)
                             has-code? (re-find phone-code-regex value)]
                         (cond
                           (not has-prefix?)
                           :errors/phone-invalid-prefix
                           (not has-code?)
                           :errors/phone-invalid-country-code
                           :else
                           :errors/phone-invalid-number)))}
                   phone-regex])
#2020-05-3103:39naomarikMaybe would be helpful in the tips section 🙂#2020-05-3104:03naomarikActually this doesn't work in clojurescript. (m/properties schema) returns #object[:cljs.coreMetaFn] and don't know what to do with that.#2020-05-3104:32naomarikI noticed (sci.core/eval-string "(inc 1)") was returning the same MetaFn object and not 2 like the sci docs says it should. Just tried using the latest sci dependency and the error/fn works as expected. @ikitommi would you expect any issues using the latest sci with the current implementation of malli? Will do more testing on my end, I'm using the exact same validations for frontend and backend.#2020-05-3105:56ikitommi@naomarik I would expect latest sci works ok.#2020-05-3106:04ikitommiAlso, if you define the validations as cljc code, e.g. no need to serialize over the wire, you can use plain functions instead of fn source code: (defn valid? [x] (and (int? x) (> x 10))) `(m/validate [:fn valid?] 11) ; => true` #2020-06-0218:37naomarikHey @ikitommi, I have all my malli specs in a cljc file, already. If I pass in functions instead of quoted functions, the clojurescript side won't through SCI will it? If that's the case will sci dependency get removed with advanced builds?#2020-06-0307:16naomarik@ikitommi Sorry was being lazy about this didn't try it yet... Just tested myself and SCI adds about 0.5 megs to the (non gzipped) output and 93kb to the final JS output with gzip -9. I'm not sending code to be evalled as string in my project, is it cool if we make SCI an optional require?#2020-06-0307:18naomarikI can try to do that myself but not sure how clean my solution would be compared to what you can come up with. Let me know what you think. I think I could also help with the docs to state that if sending functions over the wire is being used then SCI will be included for a heavier JS file.#2020-06-0307:20naomarikI think this function
(defn ^:no-doc eval [?code]
  (if (fn? ?code) ?code (sci/eval-string (str ?code) {:preset :termination-safe
                                                      :bindings {'m/properties properties
                                                                 'm/name name
                                                                 'm/children children
                                                                 'm/map-entries map-entries}})))
if we conditionally require sci if the if test fails the fn? check would be good.
#2020-06-0307:50naomarikJust tried, conditional requires seem to not work, not sure what I was expecting requiring a library at runtime :)#2020-05-3106:05ikitommithere could also be an option to add custom bindings to sci, so one could use own fn's as part of the sci source code.#2020-05-3106:09ikitommibasically just merge to existing bindings https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L878-L883#2020-05-3110:41naomarikYeah adding to the list of sci bindings would have been the most obvious thing to do for me . I saw it wasn't parameterized though so kept investigating. Actually I didn't try the error/fn property without quoting it, passing in a function is a lot better in my use case. All the examples show quoted functions in the github page, so didn't think to try that.#2020-05-3111:47ikitommiDoc updates welcome#2020-06-0213:31ikitommiGreat talk by @plexus about malli: https://youtu.be/ww9yR_rbgQs#2020-06-0213:31ikitommiand the event today: https://clojureverse.org/t/new-online-meetup-clojure-european-summer-time/6018/6#2020-06-0214:02katoxJust a thought: one thing that makes debugging Malli structures hard is that it doesn't retain any information of how it was composed. If you def a SubPart and then def a Part based on that using mu/union that info is lost. Not that it would always be possible (because schemas can just be created programatically) but it might be useful to stuff something into Meta to display it later in the explain.#2020-06-0415:20Vincent CantinIn Minimallist, there are :let and :ref nodes which make composition of models explicit. Maybe Malli will get something similar once it supports recursion.#2020-06-0718:33ikitommiArne had a custom :ref too.#2020-06-0718:34ikitommithere is also some thinking in the README about entities & values: https://github.com/metosin/malli#entities-and-values#2020-06-0718:34ikitommiHi all. Goal is to get malli out of pre-alpha before my summer vacations = this month.#2020-06-0718:35ikitommistarted with adding some new Protocols: https://github.com/metosin/malli/pull/204#2020-06-0806:44ikitommifirst class :string coming up:
(m/validate :string "")
; => true

(m/validate [:string {:min 1}] "")
; => false

(json-schema/transform [:string {:min 1, :max 4}])
; => {:type "string", :minLength 1, :maxLenght 4}

(mg/sample [:string {:min 1, :max 4}])
; => ("RMmR" "5" "oL" "G7" "ENo" "cAh" "iOb" "jG" "l8" "kuo")

(-> [:map
     [:a :string]
     [:b [:string {:min 1}]]
     [:c [:string {:max 4}]]
     [:d [:string {:min 1, :max 4}]]]
    (m/explain
      {:a 123
       :b ""
       :c "invalid"
       :d ""})
    (me/humanize))
;{:a ["should be string"],
; :b ["should be at least 1 characters"],
; :c ["should be at most 4 characters"],
; :d ["should be between 1 and 4 characters"]}
#2020-06-0807:08ikitommihttps://github.com/metosin/malli/pull/205#2020-06-0807:37naomarikPerfect. I accidentally had a string? predicate in my spec awhile ago allowing unbounded inputs.#2020-06-0809:41pithyless@U055NJ5CC any chance for a blank? option? Or is that too specific to add to malli-core? It's the first thing I have to add to all my string specs, since you almost never want to allow user input of just blank characters. In a similar vein, do you think malli-core generator should be using char-alphanumeric or simply string? I understand the first gives you nicer looking example data, but the second is more useful in testing edge cases.#2020-06-0810:06pithylessOn second thought, the latter question is probably not worth the hassle (and one can always write a custom gen where it's needed). But still curious about a possibility of blank? or present? option.#2020-06-0810:07pithylessI moved my question to the PR, where it is probably more appropriate.#2020-06-0906:43ikitommi@U05476190 chaned the generator to char & string.#2020-06-0818:50ikitommiExample string-trimmer here too:
(require '[malli.transform :as mt])
(require '[malli.core :as m])
(require '[clojure.string :as str])

(defn string-trimmer []
  (mt/transformer
    {:decoders
     {:string
      {:compile (fn [schema _]
                  (let [{:string/keys [trim]} (m/properties schema)]
                    (when trim #(cond-> % (string? %) str/trim))))}}}))

(m/decode [:string {:min 1}] "    " string-trimmer)
; => "    "

(m/decode [:string {:string/trim true, :min 1}] "    " string-trimmer)
; => ""

(m/decoder :string string-trimmer)
; => #object[clojure.core$identity] ... no-op! :)
#2020-06-0906:13ikitommiAs the renaming of m/name to m/type got only upvotes, going to change that: https://github.com/metosin/malli/pull/206. It’s a BREAKING CHANGE - both for the public api & for people who have extended the Schema protocol. Sorry for that.#2020-06-0907:09ikitommiMerged the protocol extensions (https://github.com/metosin/malli/pull/204) and realized that there is a breaking change, malli.generator/-generator is used by the new malli.generator/Generator protocol, so the old multimethod is called malli.generator/-schema-generator. Feedback welcome.#2020-06-0907:27ikitommistarted to track the before-release changed into https://github.com/metosin/malli/blob/master/CHANGELOG.md#unreleased, as some are breaking, for the better.#2020-06-1107:22pithylessI'm struggling to wrap my head around custom transformers and validators; is there some way to create a pipeline of validate -> transform -> validate (ala spec/conform ?) Or when defining a transform interceptor, can I in some way tell the the chain to stop executing further since some assumption is invalid?#2020-06-1110:41ikitommicurrently no way to stop the chain in interceptors @pithyless, could be added easily with reduced. There are Schemas that short-circuit in -transform like the :or.#2020-06-1110:42ikitommiin reitit-malli, there is a Corcion protocol and impl which does transform + validate for requests and transform + validate + transform for responses.#2020-06-1110:42ikitommifor responses, it’s basically doing: 1. apply defaults, strip extra keys 2. validate the result 3. encode to whatever (string, json, xml)#2020-06-1110:43ikitommiIf you have some minimal sample, could take a look.#2020-06-1111:23pithyless^ the reitit Coercer case is the kind of thing I have in mind. I've been going back and forth on this, but malli has been forcing me to rethink my approach: perhaps, I should just assume there is only one canonical form of data inside my project runtime, and only at the boundaries do all the encoding/decoding via custom protocols (i.e. a custom Coercer protocol and implementation for reitit, datomic, etc.)#2020-06-1111:33ikitommithe reitit code is here: https://github.com/metosin/reitit/blob/master/modules/reitit-malli/src/reitit/coercion/malli.cljc. Basically, at route(r) creation time, for each Schema, a reified Coercion is created, storing encoder, decoder, validator and explainer for that schema. At runtime, it’s just a calling those (optimized fns) in whatever order suits for the use case.#2020-06-1206:10ikitommi#2020-06-1206:10ikitommiinitial bundle-size anaysis for cljs. related to #203#2020-06-1209:36borkdude@ikitommi Note that about 200kb of the sci bundle is coming from docstrings 🙂#2020-06-1209:37borkdudeat least, if you bump to the newest sci#2020-06-1210:07ikitommiyes, updated the latest. Haven’t really studied how the dce works and how can libraries be smaller. Would be great if one could drop off whole protocol method impls if the protocol method was not used. E.g. if you don’t have any calls to the transformers, we coud remote the -transformer methods, which is easily 30% of the whole malli.core.#2020-06-1210:07ikitommimost of the concerns are in separate namespaces (json-schma, human errors, inferring), but the important ones are in the core right now. But not always important 🙂#2020-06-1210:08ikitommialso, would be great to have a sci-liteversion without the docstrings.#2020-06-1210:09borkdudemaybe that can be accomplished using a goog/define#2020-06-1210:10borkdude
:closure-defines {"sci.foo/DOCSTRINGS" false}
#2020-06-1210:11borkdudemade an issue for it: https://github.com/borkdude/sci/issues/352#2020-06-1407:17ikitommiGoing to change how the registry is composed. Goal is to allow easy bootstrapping of custom registries. Allows much smaller bundle size on cljs and one can (hopefully) easily add custom registries.#2020-06-1407:21ikitommifew sample sizes (gzipped), for malli.core (just validation): • current (full): 12kb • just :map , :vector and :string: 4kb • just :string: 700b#2020-06-1407:25ikitommiI was surprised that the closure dce actually removes all the protocol method impls too if there is nothing using those methods. e.g. not using m/decode or m/encode => all -transformer impls are removed under :advanced. This is really, reaaly, great.#2020-06-1407:26ikitommiapp that uses all the features from malli, might still be big, e.g. validation, human errors, transformation, programming with schemas etc.#2020-06-1418:36naomariknice!#2020-06-1418:36naomarikone step closer to malli frontend domination#2020-06-1610:45ikitommi#2020-06-1610:46ikitommiwith default, malli is all-immutable, with a CLOSURE_DEFINES or JVM Prop setup, the default registry can be swapped out in the user space.#2020-06-1610:47ikitommiallows also really smaller bundle sizes, moved all the defs into defns so closure can eliminate those if needed.#2020-06-1610:48ikitommi@plexus, would that help with your acme-schema sample?#2020-06-1610:48ikitommialso, the options would be default (immutable) or managed (swappable). any suggestions for better names?#2020-06-1611:29plexusyeah I think that would help us#2020-06-1614:21ikitommi#2020-06-1614:22ikitommieven more evil. optional MUTABLE registry constructor, just pass in your own atom.#2020-06-1616:04Vincent Cantinwould it be possible to avoid having any mutable registry by having the user define his own API functions defaulting to their own registry?#2020-06-1616:05Vincent Cantinmaybe via a macro? (def-malli-api my-registry)#2020-06-1617:03ikitommithat is one approach, and can be done on top of the current api. Not a fan of macros generating functions personally. Need extra work to work with static analysis tools (like cursive) Also, the api should require all the malli namespaces to be complete.#2020-06-1617:12ikitommiI'll make set-default-regisyry! to throw if the registry swapping is not enabled.#2020-06-1804:17ikitommimerged the registries in.#2020-06-1804:18ikitommiquick poke on the recursive schemas, using :ref:
(def ConsCell
  [:maybe {:id :cons}
   [:tuple int? [:ref :cons]]])

(m/explain ConsCell [1 [2 [3 [4 nil]]]])
; => nil

(m/explain ConsCell [1 [2 [3 [4]]]])
;{:schema [:maybe {:id :cons} [:tuple int? [:ref :cons]]],
; :value [1 [2 [3 [4]]]],
; :errors (#Error{:path [2 2 0 2 2 0 2 2 0 2],
;                 :in [1 1 1],
;                 :schema [:tuple int? [:ref :cons]],
;                 :value [4],
;                 :type :malli.core/tuple-size})}
#2020-06-1805:18viestiNice! :)#2020-06-1805:25Vincent Cantindoes it still work with cross recursion ? ping refers to pong, and pong refers to ping#2020-06-1804:51ikitommiok, need to figure out how things like -accept should work with recursive schemas as it’s eager (validation and explain can be both lazy). I guess need to keep a local book keeping not to walk over already walked :refs so that it doesn’t go to infinite loop.#2020-06-1805:23Vincent Cantin@ikitommi welcome to the club of infinite loop avoiders :-)#2020-06-1805:45ikitommidoes not work yet with mutual recursion, but I think we need :registry schema anyway, like there is :definitions for JSON Schma. Needed anyway for persisting the things. maybe something like:
(explain
  [:registry
   {:schemas [[:maybe {:id :ping} [:tuple [:eq "ping"] [:ref :pong]]]
              [:maybe {:id :pong} [:tuple [:eq "pong"] [:ref :pong]]]]}
   [:ref :pong]]
  ["ping" ["pong" ["ping" nil]]])
; => true
#2020-06-1805:47ikitommibtw, a colourful discussion at the old draft PR: https://github.com/metosin/malli/pull/117#2020-06-1816:07pithylessI've got
:closure-defines  {malli.registry/type "custom"}
in my shadow-cljs.edn, but still seeing error:
can't set default registry {:type "default"}
Any ideas what I may be forgetting?
#2020-06-1816:10pithylessoh, that's weird - some kind of caching bug; forcing a recompile (after it showed the error) and it went away. ¯\(ツ)/¯#2020-06-1816:17pithylessThe new registry setup allows for easier bootstrapping custom code. :thumbsup:#2020-06-1817:14jkentit doesn’t look like the new first class :string is bundled with the latest reitit [metosin/reitit-malli "0.5.2"] is there reason for this or am I missing something?#2020-06-1818:02pithylessmetosin/reitit-malli on clojars is dependent on malli 0.0.1-SNAPSHOT - theoretically, last version was uploaded yesterday, but maybe you have some caching issues? (I think clojure deps caches any snapshot locally for 24(?) hours). Perhaps try to update your deps to depend directly on latest malli from master?#2020-06-1908:04pithylessI think there may be some kind of race-condition bug with the malli.registry/type code in closure-defines. Once in a while (after a clean shadow restart) I see the {:type "default"} error, but it works if I "force compile" in shadow.#2020-06-1908:24ikitommi@pithyless that is weird, might be a bug in #shadow-cljs ?#2020-06-1908:27ikitommi@jkent I'll release a new reitit version next week, master is build against the latest malli.#2020-06-1918:25eskosDefault update policy for snapshots in pomegranate is daily; that is, once per 24 hours. If you're using Leiningen, see https://github.com/technomancy/leiningen/blob/stable/sample.project.clj#L115-L117 and if you're using something else, I have no idea 🙂#2020-06-1918:26eskosOh that was weird, got a whole screen of text after typing that... Slack's lagging. Meant this as continuation to @pithyless#2020-06-2019:23dcjA month or so ago I used Malli for a project, and with recent changes to Malli, my code broke:
(def predicate-registry
  (-> m/predicate-registry
      (-register-var #'zoned-date-time?)
      (-register-var #'local-date?)))


(def registry
  (merge predicate-registry m/class-registry m/comparator-registry m/base-registry))
So now the -registry is -schemas, but then my -register-var calls fail. I'll keep working on this, but any ideas?
#2020-06-2020:03dcjOK, this worked:
(def registry
  (merge (m/predicate-schemas)
         (m/class-schemas)
         (m/comparator-schemas)
         (m/base-schemas)
         {:zoned-date-time (m/fn-schema :zoned-date-time #'zoned-date-time?)
          :local-date      (m/fn-schema :local-date      #'local-date?)}))
And then I changed my schema references from zoned-date-time? to :zoned-date-time, etc, and also changed the associated transformer encoder/decoder names
#2020-06-2020:30ikitommi@dcj we are doing final code polishing to get the non-pre-aloha out. The breaking changes are listed in https://github.com/metosin/malli/blob/master/CHANGELOG.md#unreleased. Also, if you are using custom schemas, you should read https://github.com/metosin/malli/blob/master/README.md#schema-registry. Lot of options, and supports properly dce on cljs now#2020-06-2207:02ikitommiAbout recursive schemas. It seems that because of those, should to to add a (-children [this]) method into Schema protocol. Why? Currently, there is generic children function that returns the Schema AST for the children. The AST is unaware of any instance bindings such as local recursion targets. Given a Schema:
[:maybe {:id :cons}
   [:tuple int? [:ref :cons]]]
, the generator for :tuple just sees children of (int? [:ref :cons])and doesn’t know what :cons refers to and fails.
#2020-06-2207:05ikitommiWe could hack around this by pulling out options that created :tuple and using those to re-create the child schemas, providing all the needed local context. But, this is error-prone as one needs to wire-up all generators, visitors etc. using the original options from the father schema.#2020-06-2207:05ikitommi(currently passing the local context via options, which seems to be a good way to do it)#2020-06-2207:06ikitommiAnyway, this works now:#2020-06-2207:06ikitommi#2020-06-2207:07ikitommiwith recursion, one can set the recursion limit, just like with spec. One setting, but each :ref is counted separatly, defaulting to 10.#2020-06-2207:58ikitommiwrote more to the recursion issue, comments welcome: https://github.com/metosin/malli/pull/117#issuecomment-647348039#2020-06-2214:06Vincent CantinI feel that having more than 1 way of defining recursive structures implies that users will have to choose which to use. Choice can sometimes be a problem more than a solution.#2020-06-2305:45ikitommimutual recursion, current status:
(mg/generate
  ::ping
  {:registry (mr/composite-registry
               (m/default-schemas)
               {::ping [:maybe [:tuple [:= "ping"] [:ref ::pong]]]
                ::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]})
   :size 7, :seed 86})
; => ["ping" ["pong" ["ping" ["pong" ["ping" nil]]]]]
tested also a fail-fast on ambiguity with refs, so instead of:
(mg/generate
  ::ping
  {:registry (mr/composite-registry
               (m/default-schemas)
               {::ping [:maybe {:id ::pong} [:tuple [:= "ping"] [:ref ::pong]]]
                ::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]})
   :size 7, :seed 86})
; => ["ping" ["ping" ["ping" ["ping" ["ping" nil]]]]]
the default code will throw instead:
Execution error (ExceptionInfo) at malli.core/fail! (core.cljc:80).
:malli.core/ambiguous-ref {:type :ref, :ref :user/pong}
#2020-06-2305:46ikitommithe code is in a new PR: https://github.com/metosin/malli/pull/209#2020-06-2306:48Vincent Cantin@ikitommi [suggestion] if you can embed registries locally, then maybe you don't need to have support for a custom global registry - just ask the users to include their registry as a part of their models.#2020-06-2306:59ikitommiThe custom Schema elements can’t be embedded as data, if they are code. It’s good have a mechanism to add those to the schema registry. All user/project-defined that are just data COULD be used via embedded registries, a decision done in user space.#2020-06-2307:02ikitommiI woudn’t register any project-spesific (data) schemas into global registry, as they can be references mosty as Vars.#2020-06-2307:05ikitommiThe recursive schemas could also be allowed to be introduces using Vars, but not sure if that’s a good idea.#2020-06-2307:06ikitommiSomething like:
(declare Pong)

(def Ping [:maybe [:tuple [:= "ping"] [:ref #'Pong]]])

(def Pong [:maybe [:tuple [:= "pong"] [:ref #'Ping]]])
#2020-06-2307:09ikitommiImaginary example with not much boilerplate with schematized fns using custom schema element :db/ref registered into the global registry:
(m/defn my-fn [x :- int?, y :- [:maybe [:db/ref uuid?]]
  (println x y))
#2020-06-2307:14ikitommiAbout the Var refs - how would that be serialized? It could be done by having an asymmetric m/form for that component, which might be a bad idea:
(m/form [:maybe [:tuple [:= "ping"] [:ref #'Pong]])
;[:registry
; {:registry {::ping [:maybe {:id ::pong} [:tuple [:= "ping"] [:ref ::pong]]]
;             ::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]}}
; [:ref ::ping]]
#2020-06-2307:14ikitommior is it?#2020-06-2307:47ikitommithis works now too:
(mg/generate
  [:registry
   {:registry {::ping [:maybe [:tuple [:= "ping"] [:ref ::pong]]]
               ::pong [:maybe [:tuple [:= "pong"] [:ref ::ping]]]}}
   [:ref ::ping]]
  {:size 7, :seed 86})
; => ["ping" ["pong" ["ping" ["pong" ["ping" nil]]]]]
#2020-06-2507:28ikitommilocal registries, holes, nesting and masking - the recent thoughts from the hammock, comments most welcome: https://github.com/metosin/malli/pull/117#issuecomment-649305848#2020-06-2811:23ikitommiAbout to add eager references too:
(def Schema
  [:and
   {:registry {::a ::b
               ::b ::c
               ::c [:schema pos-int?]}}
   [:and ::a ::b ::c]])

(m/form Schema)
;[:and {:registry #:user{:a :user/b
;                        :b :user/c
;                        :c [:schema pos-int?]}}
; [:and :user/a :user/b :user/c]]

(m/to-map-syntax Schema)
;{:type :and
; :properties {:registry #:user{:a :user/b
;                               :b :user/c
;                               :c [:schema pos-int?]}}
; :children [{:type :and
;             :children [{:type :schema
;                         :children [:user/a]}
;                        {:type :schema
;                         :children [:user/b]}
;                        {:type :schema
;                         :children [:user/c]}]}]}
#2020-06-2811:25ikitommie.g. each registry hop retains the information the original linkage + :schema element to mark Entitys in a schema data graph.#2020-06-2811:25ikitommi(all registry hops are internaly :schema too)#2020-06-2811:26ikitommiaccept can be configured to automatically walk over those, if wanted.#2020-06-2811:26ikitommiboth :schema and :ref also implement the new
(defprotocol RefSchema
  (-deref [this] "returns the referenced schema"))
#2020-06-2812:16ikitommi… and for the given schema, the validation fn is just pos-int?#2020-06-2812:16ikitommi
(m/validator Schema)
; #object[clojure.core$pos_int]
#2020-06-2812:16ikitommiwhich is kinda awesome 🙂#2020-06-3010:00borkdude@ikitommi https://github.com/metosin/malli/pull/210#2020-06-3010:03borkdudeI made a PR to malli which makes sci optional. The tests are failing, but this is only because the order in which namespaces are loaded is random. sci.core needs to be loaded first. But since sci.core is now optional, the tests have to be refactored accordingly anyway. Just putting this here in case someone has ideas about it.#2020-06-3010:03borkdudeLocally the tests pass, FWIW.#2020-06-3010:19borkdudeI might put that dynaload code into a library so it can be used in more projects.#2020-06-3012:49ikitommiwill check that out, thanks!#2020-06-3012:49ikitommia separat one-purpose lib would be good#2020-06-3012:49ikitommimerged recursion PR into master.#2020-06-3012:50ikitommihere’s a sample (the generation is blocking the main thread, should port to use web-worker?): https://malli.io/?value=%7B%3Alines%20%5B%7B%3Aburger%20%7B%3Aname%20%22NAUGHTY%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Adescription%20%22Finnish%20100%25%20beef%20patty%2C%20cheddar%2C%20St%20Agur%20blue%20cheese%2C%20bacon%20jam%2C%20rocket%2C%20aioli%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Aorigin%20%7B%3Aname%20%3AFI%7D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Aprice%2011%7D%2C%0A%20%20%20%20%20%20%20%20%20%20%3Aamount%202%7D%5D%2C%0A%20%3Adelivery%20%7B%3Adelivered%20false%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%3Aaddress%20%7B%3Astreet%20%22H%C3%A4meenkatu%2010%22%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Azip%2033100%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Acountry%20%7B%3Aname%20%3AFI%2C%20%3Aneighbors%20%5B%7B%3Aname%20%3APO%7D%5D%7D%7D%7D%7D&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%3Auser%2Fcountry%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20%5B%3Aenum%20%3AFI%20%3APO%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aneighbors%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aoptional%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Avector%20%5B%3Aref%20%3Auser%2Fcountry%5D%5D%5D%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Fburger%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20string%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Adescription%20%7B%3Aoptional%20true%7D%20string%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorigin%20%5B%3Amaybe%20%3Auser%2Fcountry%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprice%20pos-int%3F%5D%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Forder-line%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aburger%20%3Auser%2Fburger%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aamount%20int%3F%5D%5D%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Forder%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Alines%20%5B%3Avector%20%3Auser%2Forder-line%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Adelivery%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%7B%3Aclosed%20true%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Adelivered%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aaddress%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Astreet%20string%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Azip%20int%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acountry%20%3Auser%2Fcountry%5D%5D%5D%5D%5D%5D%7D%7D%0A%20%3Auser%2Forder%5D#2020-06-3015:28borkdude@ikitommi https://github.com/borkdude/dynaload#2020-06-3016:43SathiyaHi. I have an use case with the data structure and scenario similar to the following example { "name": string, "age": int, "married": boolean, "sex": string "partner": { "husband":{"name": string, "age": int}, "wife":{"name": string, "age": int} } } name in husband is mandatory if the status is married and sex is Male. name in wife is mandatory if status is married and sex is female. both are not needed if married is false. how do i write the malli schema validations for the following scenario. if both the fields are at the same level i could add a function and validation. since one is at a parent and the other field is inside the child, i am unable to find a solution to implement this. any help would be appreciated. thanks in advance#2020-06-3017:29ikitommihi @kspriyan31. You can add rules to top-level and add error metadata to point into nested fields. Something like:
(def Person
  [:and {:registry {:partner [:map ["name" string?] ["age" int?]]}}
   [:map
    ["name" string?]
    ["age" int?]
    ["married" boolean?]
    ["sex" string?]
    ["partner" {:optional true}
     [:map
      ["husband" {:optional true} :partner]
      ["wife" {:optional true} :partner]]]]
   
   [:fn {:error/message "wife name is mandatory for married men"
         :error/path ["partner" "wife" "name"]}
    (fn [{:strs [sex married partner]}]
      (not (and married (= "male" sex) (not (get-in partner ["wife" "name"])))))]
   
   [:fn {:error/message "husband name is mandatory for married female"
         :error/path ["partner" "wife" "name"]}
    (fn [{:strs [sex married partner]}]
      (not (and married (= "female" sex) (not (get-in partner ["husband" "name"])))))]])

(-> Person
    (m/explain {"name" "Mauno"
                "age" 31
                "married" true
                "sex" "male"
                "partner" {"wife" {"age" 41}}})
    (me/humanize))
;{"partner" {"wife" {"name" ["missing required key" 
;                            "wife name is mandatory for married men"]}}} 
#2020-06-3017:34ikitommihere’s the same in http://malli.io: http://shorturl.at/hmuFV#2020-06-3017:38SathiyaThank you @ikitommi#2020-06-3017:39ikitommiif you change the sex to [:enum "male" "female" "other"]the sample value generation can generate better values:
({"name" "",
  "age" 1,
  "married" true,
  "sex" "male",
  "partner" {"husband" {"name" "", "age" -1}, "wife" {"name" "", "age" 0}}}
 {"name" "", "age" 2, "married" false, "sex" "male"}
 {"name" "K5", "age" 2, "married" false, "sex" "male", "partner" {"wife" {"name" "", "age" -1}}}
 {"name" "", "age" 3, "married" false, "sex" "male"}
 {"name" "8Mn",
  "age" 3,
  "married" true,
  "sex" "male",
  "partner" {"husband" {"name" "J", "age" -4}, "wife" {"name" "", "age" 3}}}
 {"name" "bsyt4", "age" 1, "married" false, "sex" "female"}
 {"name" "8k5yk", "age" 6, "married" false, "sex" "female", "partner" {"husband" {"name" "4", "age" 13}}}
 {"name" "5ut0D2sm",
  "age" 1,
  "married" true,
  "sex" "female",
  "partner" {"husband" {"name" "", "age" -102}, "wife" {"name" "i4Q5l6xa", "age" 0}}}
 {"name" "", "age" 1, "married" false, "sex" "male"}
 {"name" "l", "age" 3, "married" false, "sex" "male", "partner" {"husband" {"name" "Ixh4drJNb", "age" -37}}})
#2020-06-3017:39ikitommiyour welcome#2020-07-0120:47katoxI'm looing at https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj How can I enable the same kind of coercion but to path-params instead of query params? If I change just the key it goes through untouched.#2020-07-0123:34jkent@katox here’s an example of path coercion
["/foo/:x/:y"
     {:get {:summary    "plus with malli query parameters"
            :parameters {:path [:map [:x int?] [:y int?]]}
            :handler    (fn [{{{:keys [x y]} :path} :parameters}]
                          {:status 200
                           :body   {:total  (+ x y)}})}}]
#2020-07-0208:02katox@jkent of course, a shorter key than variants I tried. Thank you, it works!#2020-07-0210:50ikitommiAlmost there:
(def Order
  [:schema
   {:registry {"Country" [:map
                          [:name [:enum :FI :PO]]
                          [:neighbors [:vector  [:ref "Country"]]]]
               "Burger" [:map
                         [:name string?]
                         [:description {:optional true} string?]
                         [:origin [:maybe "Country"]]
                         [:price pos-int?]]
               "OrderLine" [:map
                            [:burger "Burger"]
                            [:amount int?]]
               "Order" [:map
                        [:lines [:vector "OrderLine"]]
                        [:delivery [:map
                                    [:delivered boolean?]
                                    [:address [:map
                                               [:street string?]
                                               [:zip int?]
                                               [:country "Country"]]]]]]}}
   "Order"])

(malli.mermaid/class-diagram Order)
; classDiagram
;   class Country {
;     :name [:enum :FI :PO]
;     :neighbors [:vector #:gen{:max 0} [:ref Country]]
;   }
;   class Burger {
;     :name string?
;     :description string?
;     :origin [:maybe Country]
;     :price pos-int?
;   }
;   class OrderLine {
;     :burger Burger
;     :amount int?
;   }
;   class Order {
;     :lines [:vector OrderLine]
;     :delivery Order_Delivery
;   }
;   class Order_Delivery_Address {
;     :street string?
;     :zip int?
;     :country Country
;   }
;   class Order_Delivery {
;     :delivered boolean?
;     :address Order_Delivery_Address
;   }
;   Country o-- Country
;   Burger o-- Country
;   OrderLine o-- Burger
;   Order o-- OrderLine
;   Order *-- Order_Delivery
;   Order_Delivery_Address o-- Country
;   Order_Delivery *-- Order_Delivery_Address
#2020-07-0210:50ikitommi#2020-07-0210:53ikitommias the map keys can be anything, the registry keys can be any non-vector. Strings looks nice.#2020-07-0210:58ikitommiwhen a Schema is created (using m/schema), all the registry refs are eagerly validated and the current values are captured: • have an Invalid ref? -> fail early • using mutable registry? -> the created schema instance (including refs!) is still immutable • wan’t to know the local registry which the schema was created with -> accessible via m/options#2020-07-0211:00ikitommiclojure.spec with mutable global registry, fail late and mutable refs doesn’t seem that right anymore.#2020-07-0219:50kwrooijenIs this intended behavior? Here we say that the :id field is optional in the map schema. Which works
(m/validate
 [:map
  [:id {:optional true} int?]]
 {}) ; => true
Here we give the :id field a default value. But this doesn't work
(m/decode
 [:map
  [:id {:default 42} int?]]
 {}
 mt/default-value-transformer) ; => {}
Instead, we have to give field's spec a default, not the map's key
(m/decode
 [:map
  [:id [:and {:default 42} int?]]]
 {}
 mt/default-value-transformer) ; => {:id 42}
Personally I find the last example a bit strange (Or better yet, I find it strange that the second doesn't work). I'm not saying the ints have a default, I'm saying that this maps :id field has a default. Just like how :id is an optional field, not that that int? is an optional schema. Maybe there's a reason it's designed this way? I could just be looking at this from the wrong angle.
#2020-07-0219:52kwrooijenI guess it's more that I find it very unexpected that the second example doesn't work#2020-07-0307:27ikitommi@kevin.van.rooijen handling map-entry properties is the last issue to be resolved to make the malli design complete for release, see https://github.com/metosin/malli/issues/116#2020-07-0307:29ikitommiHave some ideas how to fix that elegantly, but comments on the issues most welcome#2020-07-0307:29kwrooijenAh ok, good to hear#2020-07-0307:29kwrooijenI did some digging and realized I asked the same question 3 months ago haha (but no answer at that time)#2020-07-0307:30ikitommiHmm.. could you link that with #86?#2020-07-0307:31ikitommior do you mean here? Slack history kinda sucks#2020-07-0307:31kwrooijenApparently I asked that question in my PR to support qualified keywords in maps#2020-07-0307:32kwrooijenI had a bit of a monologue in that pr haha#2020-07-0307:33kwrooijenhttps://github.com/metosin/malli/pull/194#issuecomment-609977835#2020-07-0307:34ikitommithere is a bunch of PRs I know are related to this, wanted to have this figured out before merging any of those in. Sorry for the lag with PRs.#2020-07-0307:35kwrooijenDon't worry about it, it's a tough problem to handle#2020-07-0311:32sudakatuxHI.#2020-07-0311:32sudakatuxSo what would be the correct type for a date LocalDateTime to be more specific#2020-07-0311:32sudakatux
:responses  {200 {:body [:vector
                                     [:map
                                      [:id int?]
                                      [:sender_id int?]
                                      [:conversation_id int?]
                                      [:created_at string?]
                                      [:state string?]]
                                     ]}      
#2020-07-0311:33sudakatuxI tried inst? and it did not work. the error shows it as a string because of the conversion. but the validation seems to run before#2020-07-0311:38sudakatux
{:id 1,
  :message "string",
  :sender_id 1,
  :conversation_id 1,
  :created_at #object[java.time.LocalDateTime 0x4cc366e7 "2020-06-30T18:50:23.695073"],
  :state "UNREAD"}
#2020-07-0312:02ikitommihttps://github.com/metosin/malli/issues/49 is the issue for adding proper (Java8+ & goog.*) dates.#2020-07-0312:03ikitommibefore that, [:fn (partial instance? LocalDateTime)] does the validation part#2020-07-0312:24sudakatuxcool thanks#2020-07-0417:29ikitommistarted on thinking https://github.com/metosin/malli/pull/212#2020-07-0608:15steveb8nQ: I’m starting with Malli in my re-frame client/forms. just checking to see if there’s any gotchas/advice from the team here?#2020-07-0608:51Vincent CantinIt depends how you use Malli .. validation and coercion?#2020-07-0609:24steveb8njust validation for now. keeping it simple to start. schemas hardcoded in client code. later I intend to send schemas over the wire from the server#2020-07-0609:25steveb8nI’m using it as a validator for https://github.com/luciodale/fork in this iteration. thus no coercion since the form inputs will constrain the values#2020-07-0609:31ikitommionly notice is the bundle size, few hours away from making sci optional. Before that, the bundle is 200+kb. After, it's 1-12kb gzipped.#2020-07-0609:32ikitommioh, and you should create validators ahead of time, running m/validate can be orders of magnitude slower than just calling the returned function from m/validator.#2020-07-0609:33ikitommisame applies to m/explain vs m/explainer and to m/decode & m/decoder (and encoders, generators etc)#2020-07-0609:37ikitommifork-malli-example would be great gist/blog. Tried the first version of it, but was stumbling with nested data and for that case, did my own formik-kinda-thing for that project.#2020-07-0609:38ikitommilooking forward to seeing how the integration would look today @steveb8n !#2020-07-0609:39steveb8nthanks! great to know about the bundle increase.#2020-07-0609:39steveb8nand I’ll def the schemas for now since they are static#2020-07-0609:44steveb8nI’ll try and squeeze in a sample repo this week 🙂#2020-07-0610:13kwrooijenHey, is it possible to generate only alpha-numeric values with the :string type? Currently it will go wild and generate whatever it wants (which makes sense)#2020-07-0610:14kwrooijenOr is my only option to use a regular expression?#2020-07-0610:20ikitommi@kevin.van.rooijen you can use :gen/gen to override the generator, there is one for alphanumeric. Or use :gen/elements and list all valid in there.#2020-07-0610:21ikitommiI think :string could have format option too, like in JSON Schema. But doesn't have that yet. One format could be :alphanumeric , validator and generator would follow that...#2020-07-0610:25kwrooijenAh cool, the :gen/gen is what I need. And it would be nice to allow something like a :format option. Maybe make it extendable in some sort of way, perhaps a multimethod?#2020-07-0610:26kwrooijene.g. [:string {:format :alphanumeric}]
(defmethod malli/format :alphanumeric [_ v]
  ,,,)
Something like this, or possibly more inline with how the registry for types works
#2020-07-0701:07steveb8n@ikitommi sample repo up and running: https://github.com/stevebuik/fork-malli-ideas#2020-07-0701:07steveb8nfirst question to come from it: can the fast validation i.e. m/validator be used with m/explain?#2020-07-0704:50steveb8nand what was the “nested data” problem you had with this integration? maybe it’s another good demo in this repo#2020-07-0713:57jkentdoes malli support default error messages for enums? I’m getting "message": "unknown error" unless I do something like this for each enum:
(def source-system
  [:enum
   {:error/message "should be either: foo|bar"
    :swagger/type  "string"
    :description   "source system from where the request comes in"}
   "foo" "bar"])
#2020-07-0720:37ikitommi@jkent the default errors are defined in https://github.com/metosin/malli/blob/master/src/malli/error.cljc. There is no default for :enum, but could be. PR welcome!#2020-07-0720:38ikitommialso, if some more native english-writing person could check the spelling of the default errors, should should be int be should be an int etc…#2020-07-0802:52steveb8nhere you go….#2020-07-0802:52steveb8nhttps://github.com/metosin/malli/pull/215#2020-07-0802:53steveb8noops. broke the tests. fix coming#2020-07-0803:06steveb8nfixed!#2020-07-0721:03ikitommi@steveb8n just trying to update a value in path like [:user :address :street] , I recall it expected string paths and didn't support nesting. Might have been just user error.#2020-07-0801:25jkent@ikitommi great suggestion. here’s a PR to add a default error message for enums: https://github.com/metosin/malli/pull/214/commits#2020-07-0806:18ikitommiMerged both PRs, thanks!#2020-07-0907:36ikitommire-implemented the schema visitor using walker: m/accept is now a postwalk and and there is new m/find-first to do lazy scanning for schemas. Though of doing a prewalk too, but coudn’t find any valid use case for such and would be more complex for schema extenders to add support for it. Not sure how useful the m/find-first is and should merge this: https://github.com/metosin/malli/pull/216#2020-07-0907:39ikitommishould the m/accept be just m/walk in the future? or m/post-walk?#2020-07-0909:05ikitommi
(deftest find-first-test
  (let [schema [:map
                [:x int?]
                [:y [:vector [:tuple
                              [:maybe int?]
                              [:or [:and {:salaisuus "turvassa"} boolean?] int?]
                              [:schema {:salaisuus "vaarassa"} false?]]]]
                [:z [:string {:salaisuus "piilossa"}]]]]

    (let [walked-properties (atom [])]
      (is (= "turvassa" (m/find-first
                          schema
                          (fn [s _in _options]
                            (some->> s m/properties (swap! walked-properties conj))
                            (some-> s m/properties :salaisuus)))))
      (is (= [{:salaisuus "turvassa"}] @walked-properties)))

    (let [walked-properties (atom [])]
      (is (= "vaarassa" (m/find-first
                          schema
                          (fn [s _in _options]
                            (some->> s m/properties (swap! walked-properties conj))
                            (some-> s m/properties :salaisuus #{"vaarassa"})))))
      (is (= [{:salaisuus "turvassa"}
              {:salaisuus "vaarassa"}] @walked-properties)))))
#2020-07-0909:45alpoxHi all! Is there a possibility to merge definitions declaratively - for example in an edn definition given that the definitions to merge are defined in a registry?#2020-07-0909:48alpoxI see I can do
[:and
 [:ref :test/object]
 [:ref :test/annotated]
 [:map ....]]
but sadly this would test for all to be true rather than that later definitions override definitions from the former. Basically, I'm asking if there is something like mu/merge in a declarative way
#2020-07-0912:23ikitommi@alpox nothing just now, but sounds like a good idea. Could you write an issue of that?#2020-07-0913:39alpox@ikitommi Done: https://github.com/metosin/malli/issues/217 thanks for the response!#2020-07-0914:35euccastrowhat's the idiomatic way to specify a nonempty string in a schema? should I actually use a custom registry as in the README?#2020-07-0914:39euccastrocurrently I'm doing this
(def nonempty-string
  [:and
   string?
   [:fn {:error/message "should be nonempty string"}
    (fn [x] (pos? (count x)))]])
#2020-07-0914:42euccastroone minor problem with that is that malli/explain gives me two error messages: "should be string" and "should be nonempty string". I tried to set the :error/message in the :and instead to get only one error, but that didn't work#2020-07-0914:44euccastro(I'm using the malli version that reitit 0.5.2 requires, i.e. 0.0.1-20200525.162645-15, if that matters)
#2020-07-0914:45euccastroI'm happy to upgrade to any version that is compatible with reitit 0.5.2#2020-07-0914:58ikitommi@U65FN6WL9 [:string {:min 1}], will need to release new version to work with reitit#2020-07-0915:15euccastrothank you!#2020-07-0915:23jkent@ikitommi is there any chance you can release a new version of reitit with the latest malli?#2020-07-0917:24euccastroanother option for early adopters would be to release cutting-edge versions of malli under a separate namespace and artifact id (e.g., malli.pre-alpha) so we can use that for our own code without fear of breaking reitit. that would also reduce the pressure to update reitit unless new malli functionality is actually needed there#2020-07-0917:24euccastroplease ignore this if it sounds like too much of a hassle 🙂#2020-07-1006:22ikitommiMorning. Pushed out new immutable & compatible versions into clojars, ping @euccastro @jkent
[metosin/malli "0.0.1-20200709.163702-18"]
[metosin/reitit "0.5.3"]
#2020-07-1007:57ikitommimerged the Walker PR into master, here’s the BREAKING:
* 10.7.2020
  * `[metosin/malli "0.0.1-20200710.075225-19"]`
  * **BREAKING:**: Visitor is implemented using a Walker.
    * `m/accept` -> `m/walk`
    * `m/schema-visitor` -> `m/schema-walker`
    * `m/map-syntax-visitor` -> `m/map-syntax-walker
`
#2020-07-1207:58ikitommiAbout to replace MermaidJS with DOT, because it just works: https://github.com/metosin/malli/pull/219#2020-07-1207:59ikitommithe whole transformer is 69 loc, with most code reusable and could be part of malli.util (collecting and resolving references).#2020-07-1208:00ikitommithe actual DOT-transformer is ~30 loc 🙂#2020-07-1209:32shmuel buchnikHi I am new to malli Can I use multi with recursive ? I tried this and it does not work
(def map-multi-recursive
  [:map
   {:registry
    {::filter
     [:multi
      {:dispatch :type}
      ["in" [:map [:type [:= "in"]] [:value [:vector [:ref ::filter]]]]]
      ["or" [:map [:type [:= "or"]] [:value [:vector [:ref ::filter]]]]]
      ["not" [:map [:type [:= "not"]] [:value [:ref ::filter]]]]
      ["in" [:map [:type [:= "in"]] [:value [:map
                                             [:dimensions dimension]
                                             [:values [:vector string?]]]]]]]}}
   :filter ::filter])
#2020-07-1210:06ikitommi@shmuel.buchnic I believe the map entry should be [:filter ::filter], e.g. surround with brackets#2020-07-1210:06ikitommirecursion is a generic solution, unless there are bugs, it works with all Schemas#2020-07-1211:01shmuel buchnik@U055NJ5CC thanks for the fast response I tried and it still return an invalid schema . I will try to narrow down the issue .#2020-07-1212:59shmuel buchnik@U055NJ5CC well it is working with simple validate , but trying to use it as :coercion to body parameters and it fails .#2020-07-1213:00ikitommiyou seem to have two "in" branches in multi.#2020-07-1213:01ikitommiwill fail-fast in master: https://github.com/metosin/malli/commit/5d54e6b285c3ce440f1310302bdc2ba20a84d508#2020-07-1213:01ikitommi
(m/schema
  [:map
   {:registry
    {::filter
     [:multi
      {:dispatch :type}
      ["in" [:map [:type [:= "in"]] [:value [:vector [:ref ::filter]]]]]
      ["or" [:map [:type [:= "or"]] [:value [:vector [:ref ::filter]]]]]
      ["not" [:map [:type [:= "not"]] [:value [:ref ::filter]]]]
      ["in" [:map [:type [:= "in"]] [:value [:map
                                             [:dimensions [:tuple int? int?]]
                                             [:values [:vector string?]]]]]]]}}
   [:filter ::filter]])
; => Throws :malli.core/non-distinct-entry-keys {:keys ("in" "or" "not" "in")}
#2020-07-1213:02shmuel buchnikI fixed that the updated looks like this :
(def filter-local-registry
  {:registry
   {::filter
    [:multi
     {:dispatch :type}
     ["and" [:map [:type [:= "and"]] [:value [:vector [:ref ::filter]]]]]
     ["or" [:map [:type [:= "or"]] [:value [:vector [:ref ::filter]]]]]
     ["not" [:map [:type [:= "not"]] [:value [:ref ::filter]]]]
     ["in" [:map [:type [:= "in"]] [:value [:map
                                            [:dimension dimension]
                                            [:values [:vector string?]]]]]]]}})
#2020-07-1213:02ikitommiworks now?#2020-07-1213:03shmuel buchnikno 😞 I fixed it before u wrote forgot to update. validate call is working#2020-07-1213:03shmuel buchnikbut as :coercion it does not work#2020-07-1214:05ikitommiwhat version are you using of reitit?#2020-07-1214:07shmuel buchnik0.5.3 I debug it . It looks like at first he find filter in registry (holds 2 the default one and the local) But on later call I only see the default on the registry and than lookup fail .#2020-07-1216:18shmuel buchnikNarrowing issue : This is the schema
(def sample-request
  [:map {:registry {::age [:and int? [:> 18]]}}
   [:age ::age]])
This is the route
(def overview
  ["/api/v1/overview"
   {:swagger {:tags ["Overview"]}
    :post {:summary "get an overview data"
                   :parameters {:body sample-request}
                   :handler get-overview-data
                   }}])
The handler and other are just like in : https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj So it is not related to recursive or mutli I guess it is just wrong usage of me with registry I just did not find out what it is 😞
#2020-07-1218:22ikitommiI can reproduce this.#2020-07-1218:23ikitommiWill check this out#2020-07-1308:10ikitommifixed in https://github.com/metosin/malli/commit/9f105a53e459f9b85bfbd5c07d747fbe5d5d6f65#2020-07-1308:11ikitommipushed out [metosin/reitit "0.5.4"] with the fix in.#2020-07-1306:23steveb8nI enhanced the Malli/Fork integration demo to use idiomatic keyword maps in the re-frame app-db and string keyword for Fork forms. @lucio#2020-07-1306:23steveb8nhttps://stevebuik.github.io/fork-malli-ideas/#!/basics#2020-07-1306:23steveb8nMalli transforms makes this easy to do#2020-07-1306:24steveb8nIt has raised another question. @ikitommi when you have a chance, can you comment on the 2 questions?#2020-07-1308:17ikitommi@steveb8n sure: 1. m/explainer returns a optimized pure function for explaining, e.g:
(m/explainer [:map [:x int?]])
2. mt/transformer can be used to compose transformers, runs them in single sweep. Also, you could store the computed m/decoder :
(m/decoder
  [:map [:x int?]]
  (mt/transformer (mt/key-transformer {:decode keyword}) (mt/string-transformer)))
#2020-07-1308:19steveb8nthanks. for #2 I tried the composition via mt/transformer but couldn’t get it to transform properly. I am adding complexity by using it with a “multi-schema”#2020-07-1308:19ikitommiat best, the m/decoder (or m/encoder) know there is nothing to do and return identity:
(m/decoder
  [:map [:x int?]]
  (mt/transformer
    (mt/json-transformer)
    (mt/json-transformer)
    (mt/json-transformer)
    (mt/json-transformer)))
; => #object[clojure.core$identity]
#2020-07-1308:20steveb8nnot in the public repo, in my private app#2020-07-1308:20ikitommiIf you have and example of multi-schema that doesn’t work, please share, I could take a look#2020-07-1308:20steveb8nthanks. will do.#2020-07-1308:20steveb8nand I’ll try m/explainer as well 🙂#2020-07-1308:22ikitommitransformers are computed into interceptor chains, where the order matters.#2020-07-1308:23steveb8nI think that’s the right direction to solve it: ordering#2020-07-1308:23steveb8nbut I’ll start with the simpler (public) example#2020-07-1308:23ikitommialso, as the transformers are mounted from root schema towards leafs, the :multi decoding is applied before it’s childs (as we don’t know which child is selected), so you might need to decode the :multi dispatch key manually.#2020-07-1308:24ikitommithere is an example in the README:
(m/decode
  [:multi {:dispatch :type
           :decode/string '#(update % :type keyword)}
   [:sized [:map [:type [:= :sized] [:size int?]]]
   [:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
  {:type "human"
   :name "Tiina"
   :age "98"
   :address {:country "finland"
             :street "this is an extra key"}}
  (mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
;{:type :human
; :name "Tiina"
; :address {:country :finland}}
#2020-07-1308:24steveb8nyep: tried that too. failed. I’ll dig deeper#2020-07-1308:25steveb8nI’ll get back to you. making this work makes the fork/malli integration nice and idiomatic on both sides#2020-07-1308:25steveb8nso worth the effort from me#2020-07-1308:27ikitommioh, noticed you have the custom error message for :enum: {:error/message "Must be London or Tampere"}. Enums have something decend nowadays by default: https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L63-L66#2020-07-1308:27ikitommimaybe that could be polished into having , & or s by default?#2020-07-1308:28steveb8nok. I’ll try that too. and I’ll send a PR for the english grammar if required#2020-07-1308:28steveb8nend of the day here (Sydney) so probably not till tomorrow#2020-07-1308:51ikitommiMy guess is that you should have a new named transformer, something like:
(mt/transformer 
  {:name :before}
  (mt/key-transformer {:decode keyword}) 
  (mt/string-transformer))
and in the :multi:
[:multi {:dispatch :type
         :decode/before '#(update % :type keyword)} ...
#2020-07-1308:53ikitommithis way, the :multi selects the branch correctly before anything else, and the key-transformer & string-transformer both work correctly.#2020-07-1323:18steveb8nok. here’s a simple test case from the demo repo….#2020-07-1323:19steveb8n
(let [v {"id"               "123"
         "github-followers" "10"}
      schema [:map
              [:id :string]
              [:github-followers pos-int?]]]
  (->> (mt/transformer (mt/key-transformer {:decode keyword})
                       mt/string-transformer)
       (m/decode schema v)
       cljs.pprint/pprint))
why does the string transformer not work?
#2020-07-1408:36ikitommi@steveb8n doesn’t work :thinking_face:. Will check it out.#2020-07-1408:37ikitommi(for some reason, the keys are not followed AFTER converted to keywords.#2020-07-1408:37ikitommioh, the key-transformer applies on :leave, which is after-the-fact.#2020-07-1408:39ikitommi… should be on :enter on decode and on :leave on encode. Just a sec.#2020-07-1408:58steveb8nno hurry. the workaround of doing 2 transforms is ok. but it will be good to see the transform composition handling this use case eventually#2020-07-1408:59steveb8neven if that means dropping down to the interceptor level#2020-07-1409:00steveb8nbtw: for fun I might run https://github.com/stevebuik/Stu on my project. it’ll show the extra 200k from sci very clearly#2020-07-1409:14ikitommi@steveb8n fixed in master: https://github.com/metosin/malli/commit/679ca7780f2e36e427885e2343772369e048598e#2020-07-1409:15steveb8ngreat! that was quick. I’ll update first thing tomorrow#2020-07-1409:16steveb8n@lucio just added a feature to fork at about the same speed. you guys are awesome#2020-07-1409:16ikitommididn’t know about Stu, looks nice#2020-07-1409:17steveb8nit needs a couple of fixes to use properly. let me know if you want to use it. then I’ll go fix them#2020-07-1409:46ikitommi@shortlyportly related to you question on #reitit, would this help:
(let [path->schema (atom {})]
  (mu/find-first
    [:and
     [:fn '(constantly true)]
     [:map
      [:name string?]
      [:tags [:set [:map [:name string?]]]]
      [:address [:and
                 [:fn '(constantly true)]
                 [:map [:street string?]]
                 [:fn '(constantly true)]]]]
     [:fn '(constantly true)]]
    (fn [s i _]
      (swap! path->schema update i (fnil identity s))
      nil))
  @path->schema)
;{[] [:and
;     [:fn (constantly true)]
;     [:map
;      [:name string?]
;      [:tags [:set [:map [:name string?]]]]
;      [:address [:and [:fn (constantly true)] [:map [:street string?]] [:fn (constantly true)]]]]
;     [:fn (constantly true)]],
; [:name] string?,
; [:tags] [:set [:map [:name string?]]],
; [:tags :malli.core/in] [:map [:name string?]],
; [:tags :malli.core/in :name] string?,
; [:address] [:and [:fn (constantly true)] [:map [:street string?]] [:fn (constantly true)]],
; [:address :street] string?}
#2020-07-1409:47ikitommiyou would get the schemas per value path for standalone validation.#2020-07-1416:27Dave SimmonsThanks again @ikitommi - I'll take a look. Just wanted to say love your talks and the libraries metosin are putting out. many thanks.#2020-07-1507:54steveb8n@lucio @ikitommi I updated the demo app with the new composable transformers fix https://stevebuik.github.io/fork-malli-ideas/#!/basics#2020-07-1507:54steveb8nso now we have an example of keyword maps, validated by Malli but also transformed to string keys for Fork inputs#2020-07-1507:54steveb8nworks great!#2020-07-1508:16ikitommiadded the path->schema as malli.util/path-schemas, I think that is useful in things like generating forms from malli Schemas: you get an ordered list of value paths and the corresponding subschemas: 1. you can validate whole schema on blur / submitting 2. you can push subschema validation to components, faster local validation on keypress works for all basic clojure data structures, also for nested sequences (via :malli.core/in marker in path)#2020-07-1508:17ikitommi#2020-07-1521:21rutledgepaulvThis sounds like it might help with something I've been pondering while working on https://github.com/RutledgePaulV/ui-kit! Still working out some of the state structure / data validation / conformance stuff before I go hard on nailing down all the component types#2020-07-1521:22rutledgepaulvTrying to do minimal redraws and making sense of reagent cursors + malli trees#2020-07-1608:55ikitommi👍#2020-07-1508:18ikitommie.g. just iterate the map entries and emit form components#2020-07-1508:31steveb8nthat’s an interesting idea. this could also be used with Fork to generate forms. might have to try that 🙂#2020-07-1508:32steveb8nhow far we’ve come from multi-tenant validation!#2020-07-1807:49ikitommifinal(?) cleanup for transformers: https://github.com/metosin/malli/pull/224#2020-07-1807:49ikitommibig change is that collection transformers don't coerce their type if children don't need transformation. In practise:
(m/decode [:vector keyword?] '("abba") (mt/json-transformer))
; => [:abba]

(m/decode [:vector string?] '("abba") (mt/json-transformer))
; => ("abba")
#2020-07-1807:50ikitommifor string-transformer, works like before:
(m/decode [:vector string?] '("abba") (mt/string-transformer))
; => [:abba]
#2020-07-1807:52ikitommi=> the transformer instances can decide how to transform the collections, with JSON, both Jsonista and Cheshire decode JSON arrays as vectors, so they don’t need any “just in case” transformations.#2020-07-1807:53ikitommiif you have a deep Malli Schema, which only has things that can be presented in JSON, the transformation engine return identity. Which is nice.#2020-07-1909:08mike_ananev@ikitommi Hi. Is there any roadmap for the first release? (How many features/errors should be improved/fixed before release? )#2020-07-1910:34ikitommi@mike1452 just the https://github.com/metosin/malli/pull/212 I think. But it seems to be a rabbit hole, can’t do the error thing in it elegantly with the current api, need either to discover a simple resolution for it, to change the explain (internal) api or the LensSchema api.#2020-07-1910:35ikitommialso, checked the regal-malli integration, and might do a quick re-pacakge of the protocols so that implementing a new schema instance doesn’t require requiring all malli namespaces.#2020-07-1910:37ikitommiall small things, the first release will be alpha with 98% stabile public api and 80% stabile internal. try to get all the breaking things (for the better) before the release.#2020-07-1911:57mike_ananev👍#2020-07-2308:22ikitommisci will be an optional dependency, why? • faster standalone usage for JVM (2.5sec -> 0.5sec to load `malli.core`) • smaller js-bundles: 120kb -> 7kb first cut: https://github.com/metosin/malli/pull/227#2020-07-2308:43ikitommito make the sci-integration explicit, might be a good idea to make the whole default options customizable, instead of just the default registry. would allow one to pass sci as the default :evaluator , one could pass in default custom localizations for error messages etc. Wrote an issue here: https://github.com/metosin/malli/issues/228#2020-07-2308:44ikitommicomments welcome. it’s a rainy day, should have few hours later today to do that.#2020-07-2308:56plexusshould a schema-schema's properties delegate to the referenced schema?#2020-07-2308:56plexus
(schema/properties (malli.core/-deref (schema/schema :ars/address)))
  (schema/properties (schema/schema :ars/address))
#2020-07-2308:56plexusthe first returns a properties map, the second returns nil. would be nice if it actually returned the properties of the child schema#2020-07-2308:58plexus(I'm upgrading our project to the latest Malli. Nice addition of RefSchema and schema-schema!)#2020-07-2309:17plexusI guess the question is how transparent a RefSchema is expected to be. We had our implementation of a RefSchema before Malli added it, and in our case we tried to make it largely transparent, e.g. it implements MapSchema and LensSchema which delegate to the child schema. Was very convenient but maybe you prefer to be explicit and have (if (satisfies? RefSchema s) (-deref s) s) checks#2020-07-2309:20ikitommiGood question. :schema can now have it's own properties: [::m/schema {:title "foo"} :ars/address]#2020-07-2309:22ikitommiwould help if there was distinction between properties and derived/accumulated properties#2020-07-2309:24ikitommiproperties are used in transforming schemas, e.g. into map-format. If they are looked from child, the map-format would duplicate the properties into ::m/schema#2020-07-2309:26ikitommia new Protocol / method would solve that, but not sure if that's the right fix#2020-07-2309:45plexusmakes sense to keep them separate if they have different uses, maybe some more helpers would already go a long way, there's no deref yet for instance, only -deref. A version of deref/resolve that checks for RefSchema and otherwise returns the schema directly would also be nice.#2020-07-2309:46plexuswhat about if you do an update-in, but one of the schemas on the path is not a map schema but a reference to a map schema? should that work? (we do stuff like that and it's quite handy :))#2020-07-2313:33ikitommiMerged the SCI Optional PR. From Changelog:
* 23.7.2020
  * **BREAKING:**: `sci` is not a default dependency. Enabling sci-support:
    * **Clojure**: add a dependency to `borkdude/sci`
    * **ClojureScript**: also require `sci.core` (directly or via `:preloads`)
#2020-07-2313:33ikitommibefore and after (a small sample app)#2020-07-2313:35borkdudeWhat is the total size of the app before and after, un-gzipped? What does optimized mean, Clojure advanced?#2020-07-2313:45ikitommihere’s the javascript:
➜  malli git:(master) ✗ ls -lh app2-sci
total 1536
-rw-r--r--  1 tommi  staff   8.8K Jul 23 16:43 app.js
-rw-r--r--  1 tommi  staff   177K Jul 23 16:43 cljs.js
-rw-r--r--  1 tommi  staff    16K Jul 23 16:43 malli.js
-rw-r--r--  1 tommi  staff   2.0K Jul 23 16:43 manifest.edn
-rw-r--r--  1 tommi  staff   556K Jul 23 16:43 sci.js

➜  malli git:(master) ✗ ls -lh app2
total 288
-rw-r--r--  1 tommi  staff    10K Jul 23 16:41 app.js
-rw-r--r--  1 tommi  staff   102K Jul 23 16:41 cljs.js
-rw-r--r--  1 tommi  staff    24K Jul 23 16:41 malli.js
-rw-r--r--  1 tommi  staff   987B Jul 23 16:41 manifest.edn
#2020-07-2313:46ikitommimanually zipped:
➜  malli git:(master) ✗ ls -lh app2-sci
total 344
-rw-r--r--  1 tommi  staff   2.3K Jul 23 16:43 app.js.gz
-rw-r--r--  1 tommi  staff    35K Jul 23 16:43 cljs.js.gz
-rw-r--r--  1 tommi  staff   4.7K Jul 23 16:43 malli.js.gz
-rw-r--r--  1 tommi  staff   624B Jul 23 16:43 manifest.edn.gz
-rw-r--r--  1 tommi  staff   119K Jul 23 16:43 sci.js.gz

➜  malli git:(master) ✗ ls -lh app2
total 80
-rw-r--r--  1 tommi  staff   3.0K Jul 23 16:41 app.js.gz
-rw-r--r--  1 tommi  staff    22K Jul 23 16:41 cljs.js.gz
-rw-r--r--  1 tommi  staff   6.5K Jul 23 16:41 malli.js.gz
-rw-r--r--  1 tommi  staff   370B Jul 23 16:41 manifest.edn.gz
#2020-07-2313:46ikitommithe app itself is silly, bare-bones malli with few schemas: https://github.com/metosin/malli/blob/master/app/malli/app2.cljc#2020-07-2313:47borkdudeok#2020-07-2313:47ikitommishadow config: https://github.com/metosin/malli/blob/master/shadow-cljs.edn#2020-07-2313:48borkdudemodules are a bit confusing since one thing can get randomly moved from one module to another#2020-07-2313:48borkdudebut overall, this is a win for people who don't want to use sci#2020-07-2313:52ikitommiso total sizes: • js: 757,8kb -> 136kb • gz: 161kb -> 31,5kb , so 1/5 the size.#2020-07-2313:54ikitommiyup (javascript) is 223kB unpacked, so we can say malli can be small too. or batteries-included.#2020-07-2314:00ikitommianyone btw interested in writing a malli->typescript transformer? in a project with both, might be useful, most likely fun#2020-07-2314:01ikitommiwith yup, you can say:
type Person = yup.InferType<typeof personSchema>;
#2020-07-2314:02ikitommiI don’t think it would be much more work than the malli -> JSON Schema.#2020-07-2314:02ikitommimight be wrong 🙂#2020-07-2611:01mike_ananevHi! How I can express if A is optional in schema and if A exists then B should present too?#2020-07-2611:02borkdudeprobably using a predicate?#2020-07-2611:08mike_ananevI mean how to express it as spec (inside spec one value depends on another) ?#2020-07-2611:12mike_ananev
{
 :java-src-folder             "test-projects/javac/src/java"
 :javac-options               ["-target" "1.8" "-source" "1.8" "-Xlint:-options"] }
How to express as spec if :java-src-folder is present then :javac-options should be present?
#2020-07-2611:57ikitommi@mike1452 you can compose rules with :and, from README: `(-> [:and [:map [:password string?] [:password2 string?]] [:fn {:error/message "passwords don't match" :error/path [:password2]} '(fn [{:keys [password password2]}] (= password password2))]] (m/explain {:password "secret" :password2 "faarao"}) (me/humanize)) ; {:password2 ["passwords don't match"]}`#2020-07-2612:01mike_ananev@ikitommi Thanks! 👍#2020-07-2709:04steveb8nQ: I’m having trouble with Cursive and shadow compilation ever since I updated to the dynaload/sci update. anyone else seeing weird behaviour?#2020-07-2709:06ikitommi@steveb8n do you have the sci.core preloaded?#2020-07-2709:07borkdudeAre you using sci?#2020-07-2709:07steveb8nno but I don’t want sci in the build#2020-07-2709:07steveb8nno, not using sci#2020-07-2709:07borkdudeThen you should not preload it and it should all work fine theoretically.#2020-07-2709:07steveb8nI did previously but I don’t need it (yet) so would rather save the bundle output size#2020-07-2709:07ikitommiwhat kind of errors do you get?#2020-07-2709:08steveb8n➜ client-editor git:(master) ✗ shadow-cljs -A:dev --verbose release frontend shadow-cljs - config: /Users/steve/Documents/dev-personal/nextdoc-cloud/client-editor/shadow-cljs.edn shadow-cljs - starting via “clojure” [:frontend] Compiling ... -> build target: :browser stage: :configure <- build target: :browser stage: :configure (6 ms) -> Resolving Module: :main The required namespace “borkdude.dynaload-cljs” is not available, it was required by “malli/sci.cljc”. ➜ client-editor git:(master) ✗ clojure -Stree org.clojure/clojure 1.10.1 org.clojure/core.specs.alpha 0.2.44 org.clojure/spec.alpha 0.2.176 cljs-http/cljs-http 0.1.46 noencore/noencore 0.3.4 commons-codec/commons-codec 1.11 org.clojure/tools.namespace 0.2.11 org.clojure/core.async 0.4.474 org.clojure/tools.analyzer.jvm 0.7.0 org.clojure/tools.analyzer 0.6.9 org.clojure/core.memoize 0.5.9 org.clojure/core.cache 0.6.5 org.clojure/data.priority-map 0.0.7 org.ow2.asm/asm-all 4.2 com.lucasbradstreet/cljs-uuid-utils 1.0.2 riverford/compound 2020.01.09 appliedscience/js-interop 0.1.19 org.clojure/clojurescript 1.10.520 org.clojure/data.json 0.2.6 org.clojure/google-closure-library 0.0-20170809-b9c14c6b org.clojure/google-closure-library-third-party 0.0-20170809-b9c14c6b org.mozilla/rhino 1.7R5 com.google.javascript/closure-compiler-unshaded v20180805 com.google.jsinterop/jsinterop-annotations 1.0.0 com.google.javascript/closure-compiler-externs v20180805 com.google.guava/guava 25.1-jre com.google.errorprone/error_prone_annotations 2.1.3 org.codehaus.mojo/animal-sniffer-annotations 1.14 com.google.j2objc/j2objc-annotations 1.1 org.checkerframework/checker-qual 2.0.0 com.google.code.findbugs/jsr305 3.0.2 args4j/args4j 2.0.26 com.google.protobuf/protobuf-java 3.0.2 com.google.code.gson/gson 2.7 com.cognitect/transit-cljs 0.8.256 com.cognitect/transit-js 0.8.846 fork/fork https://github.com/luciodale/fork.git 25276c0 grafeo/grafeo 0.1.4 r0man/alumbra.printer 0.1.1 r0man/alumbra.js 0.1.0 clj-http/clj-http 3.9.1 org.apache.httpcomponents/httpasyncclient 4.1.3 org.apache.httpcomponents/httpcore-nio 4.4.6 slingshot/slingshot 0.12.2 commons-io/commons-io 2.6 org.apache.httpcomponents/httpcore 4.4.9 org.apache.httpcomponents/httpclient-cache 4.5.5 org.apache.httpcomponents/httpclient 4.5.5 commons-logging/commons-logging 1.2 potemkin/potemkin 0.4.5 clj-tuple/clj-tuple 0.2.2 org.apache.httpcomponents/httpmime 4.5.5 cheshire/cheshire 5.8.1 com.fasterxml.jackson.dataformat/jackson-dataformat-cbor 2.9.6 tigris/tigris 0.1.1 com.fasterxml.jackson.dataformat/jackson-dataformat-smile 2.9.6 re-frame/re-frame 0.10.6 org.clojure/tools.logging 0.3.1 net.cgrand/macrovich 0.2.0 reagent/reagent 0.7.0 cljsjs/react-dom 15.5.4-0 cljsjs/react 15.5.4-0 cljsjs/react-dom-server 15.5.4-0 cljsjs/create-react-class 15.5.3-0 cljc.java-time/cljc.java-time 0.1.11 cljs.java-time/cljs.java-time 0.1.16 henryw374/js-joda 1.12.0-1 metosin/malli https://github.com/metosin/malli.git 3a670d8 com.gfredericks/test.chuck 0.2.10 instaparse/instaparse 1.3.6 com.andrewmcveigh/cljs-time 0.5.1 borkdude/dynaload https://github.com/borkdude/dynaload.git 52f71bc borkdude/edamame 0.0.11-alpha.12 org.clojure/tools.reader 1.3.2 org.clojure/test.check 1.0.0 funcool/promesa 4.0.2 com.rpl/specter 1.1.3-SNAPSHOT riddley/riddley 0.1.12 metosin/reitit-frontend 0.2.9 metosin/reitit-core 0.2.9 meta-merge/meta-merge 1.0.0 metosin/reitit 0.2.9 metosin/reitit-middleware 0.2.9 metosin/muuntaja 0.6.1 com.cognitect/transit-clj 0.8.313 com.cognitect/transit-java 0.8.337 org.msgpack/msgpack 0.6.12 com.googlecode.json-simple/json-simple 1.1.1 org.javassist/javassist 3.18.1-GA javax.xml.bind/jaxb-api 2.3.0 lambdaisland/deep-diff 0.0-25 mvxcvi/puget 1.0.3 fipp/fipp 0.6.14 mvxcvi/arrangement 1.1.1 org.clojure/core.rrb-vector 0.0.13 tech.droit/clj-diff 1.0.0 metosin/reitit-swagger 0.2.9 metosin/reitit-ring 0.2.9 ring/ring-core 1.7.1 commons-fileupload/commons-fileupload 1.3.3 clj-time/clj-time 0.14.3 joda-time/joda-time 2.9.9 crypto-random/crypto-random 1.2.0 ring/ring-codec 1.1.1 crypto-equality/crypto-equality 1.0.0 metosin/reitit-schema 0.2.9 metosin/schema-tools 0.10.5 prismatic/schema 1.1.9 metosin/reitit-sieppari 0.2.9 metosin/sieppari 0.0.0-alpha6 metosin/reitit-http 0.2.9 metosin/reitit-spec 0.2.9 metosin/spec-tools 0.8.2 com.fasterxml.jackson.core/jackson-databind 2.9.7 com.fasterxml.jackson.core/jackson-core 2.9.7 com.fasterxml.jackson.core/jackson-annotations 2.9.0 metosin/reitit-swagger-ui 0.2.9 metosin/ring-swagger-ui 2.2.10 metosin/jsonista 0.2.2 org.ow2.asm/asm 5.1 virgil/virgil 0.1.6 com.fasterxml.jackson.datatype/jackson-datatype-jsr310 2.9.7 metosin/reitit-interceptors 0.2.9 camel-snake-kebab/camel-snake-kebab 0.4.0 ➜ client-editor git:(master) ✗#2020-07-2709:09steveb8nI can also make it fail by starting a clojure repl in the terminal and trying to require dynaload#2020-07-2709:09borkdudemaybe kill your ~/.gitlibs ?#2020-07-2709:10borkdudeand clear out your .cpcache#2020-07-2709:10borkdudeand use -Sforce#2020-07-2709:10steveb8nClojure 1.10.1or git:(master) ✗ clojure user=> (require ’borkdude.dynaload-clj) Execution error (FileNotFoundException) at user/eval1 (REPL:1). Could not locate borkdude/dynaload_clj__init.class, borkdude/dynaload_clj.clj or borkdude/dynaload_clj.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.#2020-07-2709:11steveb8nI did try deleting it from gitlibs but I haven’t tried deleting the whole dir#2020-07-2709:11steveb8nhave already deleted .cpcache#2020-07-2709:11steveb8ntrying -Sforce ….#2020-07-2709:12steveb8nclojure -Sforce Clojure 1.10.1 user=> (require ’borkdude.dynaload-clj) Execution error (FileNotFoundException) at user/eval1 (REPL:1). Could not locate borkdude/dynaload_clj__init.class, borkdude/dynaload_clj.clj or borkdude/dynaload_clj.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.#2020-07-2709:12borkdude@steveb8n when you do clojure -Spath do you see borkdude/dynaload there?#2020-07-2709:12steveb8nwhat’s even stranger is that it works in another project and in CI for this (problematic) project#2020-07-2709:13borkdudemaybe it's a shadow problem? can you try with vanilla CLJS?#2020-07-2709:13steveb8nyes. with -Spath it is present#2020-07-2709:13borkdudeoh you tried already with the JVM huh. that's strange#2020-07-2709:13steveb8nthat’s why I also tested using the clj repl. to exclude shadow#2020-07-2709:14steveb8nCursive is also weird. it appears in the deps panel but not in “external libraries”#2020-07-2709:14ikitommi
➜  ~ clojure -Sdeps '{:deps {metosin/malli {:git/url "" :sha "627b0f0592d129a317f8ccf3dff5296376948bf9"}}}'
Checking out:  at 627b0f0592d129a317f8ccf3dff5296376948bf9
Clojure 1.10.1
user=> (require 'borkdude.dynaload-clj)
Syntax error (ClassNotFoundException) compiling at (REPL:1:1).
'borkdude.dynaload-clj
#2020-07-2709:15borkdudewhat SHA is malli using for dynaload? maybe it was a branch that got deleted?#2020-07-2709:15borkdudethe latest SHA is 52f71bc2cb7389a932835fe02f185e3801f7e063#2020-07-2709:16steveb8nmalli is using 52….#2020-07-2709:16steveb8n@ikitommi when I run that in my terminal, it works ok#2020-07-2709:16steveb8nsorry. wrong#2020-07-2709:16steveb8nrepl starts but require fails#2020-07-2709:17steveb8nif you try the same require with the 52… sha, does it work?#2020-07-2709:17borkdude
$ clojure -Sdeps '{:deps {metosin/malli {:git/url "" :sha "627b0f0592d129a317f8ccf3dff5296376948bf9"}}}'
Cloning: 
Checking out:  at 627b0f0592d129a317f8ccf3dff5296376948bf9
Cloning: 
Checking out:  at 52f71bc2cb7389a932835fe02f185e3801f7e063
Clojure 1.10.1
user=> (require 'borkdude.dynaload-clj)
nil
user=>
#2020-07-2709:17borkdudeworks fine over here#2020-07-2709:17borkdudedoing this from a clean dir#2020-07-2709:18steveb8nclojure -Sdeps ‘{:deps {metosin/malli {:git/url “https://github.com/metosin/malli.git” :sha “627b0f0592d129a317f8ccf3dff5296376948bf9"}}}’ Clojure 1.10.1 user=> user=> user=> (require ’borkdude.dynaload-clj) Execution error (FileNotFoundException) at user/eval1 (REPL:1). Could not locate borkdude/dynaload_clj__init.class, borkdude/dynaload_clj.clj or borkdude/dynaload_clj.cljc on classpath. Please check that namespaces with dashes use underscores in the Clojure file name.#2020-07-2709:19steveb8nmust be a deps boycott on Australia#2020-07-2709:19ikitommiworking here now too :thinking_face:#2020-07-2709:19ikitommichecked out dynaload manually, started working after that.#2020-07-2709:19borkdudemaybe it's a deps.edn bug?#2020-07-2709:19steveb8nyeah I could try that#2020-07-2709:20steveb8nexplicit checkout locally#2020-07-2709:20borkdude
$ clojure -Sdescribe
{:version "1.10.1.536"
#2020-07-2709:20ikitommi
borkdude/dynaload {:git/url ""
                           :sha "52f71bc2cb7389a932835fe02f185e3801f7e063"}
#2020-07-2709:21steveb8nI’ve got 1.10.1.447#2020-07-2709:21steveb8nI’ve seen deps bugs before. I might try updating clojure cli#2020-07-2709:22borkdudeI'm willing to cut a .jar release later on, but since malli itself is still git only, I wasn't in a hurry#2020-07-2709:25steveb8nI tried adding an explicit dep on dynaload. same behaviour as with the transitive dep via malli#2020-07-2709:26steveb8nstill waiting on brew upgrade 🙂#2020-07-2709:28steveb8nare either of you using Cursive? if so, do you see dynaload in the “External Libraries”?#2020-07-2709:30steveb8nafter the clojure deps upgrade, the require works in the terminal. 1 step forward!#2020-07-2709:32steveb8nshadow compile from cli also now works. it is looking like a deps bug#2020-07-2709:32steveb8ntrying Cursive ….#2020-07-2709:34steveb8ncursive/shadow via repl still broken. I’ll keep digging…#2020-07-2709:38borkdudemaybe cursive uses its own version of deps.edn or maybe tools.deps directly which may be behind#2020-07-2709:40steveb8nthat’s exactly what is was! I switched (in prefs) to using the CLI dep executable and now shadow works inside Cursive as well#2020-07-2709:40steveb8nI got bitten by a double-whammy: deps bug and Cursive config#2020-07-2709:41steveb8nthanks for the help team 🙏#2020-07-2709:48ikitommiwhat was the root cause for the bug? I’m using 1.10.1.536 and still saw it. A stale cache?#2020-07-2709:57borkdudemaybe in your case, an AOT-cache related bug?#2020-07-2709:58steveb8nnot sure as I’ve blown away so much stuff now#2020-07-2709:58steveb8nI’m now on 1.10.1.561#2020-07-2709:59borkdudeI mean in ikitommi's case, since he had a ClassNotFoundException#2020-07-2709:59borkdudewhereas steveb8n got a file not found related exception#2020-07-2710:01steveb8nmy best guess is a bug in handling transitive deps via git deps#2020-07-2710:02steveb8nbtw: loving the reduction in bundle size. thanks both of you#2020-07-2710:02steveb8nalthough I then added vega lite and added another 200k#2020-07-2810:49dharriganAny thoughts on my PR? https://github.com/metosin/malli/pull/234. It's a shorthand way of fixing the length of the string to be accepted 🙂#2020-07-2912:48ikitommi@dharrigan not fond of that. You can already say [:string {:min 4, :max 4}] , which is not that much longer than [:string {:length 4}]. And you can always uses aliases to save on typing:
[:schema 
 {:registry {::str4 [:string {:min 4, :max 4}]}} 
 ::str4]
or:
(def string4 [:string {:min 4, :max 4}])

(m/validate string4 "kikka")
; => false
#2020-07-2912:49ikitommithere could be a pretty error formatter for that, e.g.
(and min (= min max)) (str "should be " min " characters")
#2020-07-2912:49ikitommi=>
(-> [:schema {:registry {::str4 [:string {:min 4, :max 4}]}} ::str4]
    (m/explain "kikka")
    (me/humanize))
; => ["should be 4 characters"]
#2020-07-2913:00ikitommifixed that in master, thanks for giving it a though.#2020-07-2913:02ikitommiJSON Schema also has only minLength & maxLength.#2020-07-2913:09ikitommi
(def registry
  (reduce
    (fn [acc i]
      (assoc acc (str "string" i) [:string {:min i, :max i}]))
    {}
    (range 5)))

registry
;{"string0" [:string {:min 0, :max 0}],
; "string1" [:string {:min 1, :max 1}],
; "string2" [:string {:min 2, :max 2}],
; "string3" [:string {:min 3, :max 3}],
; "string4" [:string {:min 4, :max 4}]}

(-> [:map {:registry registry}
     [:s1 "string1"]
     [:s2 "string2"]
     [:s3 "string3"]]
    (m/explain {:s1 "hi", :s2 "hi", :s3 "hi"})
    (me/humanize))
;{:s1 ["should be 1 characters"]
; :s3 ["should be 3 characters"]}
#2020-07-2915:41ikitommiIs anyone leaning on the current format of :errors of m/explain ?#2020-07-2915:43ikitommiAbout to break the :path and :in so that :path is a valid pointer to mu/get-in.#2020-07-2915:44ikitommithis gives symmetry, that should have been there since beginning.#2020-07-2915:57ikitommithe paths are so much better now 🙂
(m/explain
  [:multi {:dispatch :type}
   [:sized [:map
            [:type [:= :sized]]
            [:size [:maybe int?]]]]
   [:human [:map
            [:type [:= :human]]
            [:name string?]
            [:age int?]
            [:address [:maybe [:map [:country keyword?]]]]]]]
  {:type :human
   :name "inkeri"
   :age "100"
   :address {:country "sweden"}})
;{:schema [:multi
;          {:dispatch :type}
;          [:sized [:map [:type [:= :sized]] [:size [:maybe int?]]]]
;          [:human [:map [:type [:= :human]] [:name string?] [:age int?] [:address [:maybe [:map [:country keyword?]]]]]]],
; :value {:type :human, :name "inkeri", :age "100", :address {:country "sweden"}},
; :errors (#Error{:path [:human :age]
;                 :in [:age]
;                 :schema int?
;                 :value "100"}
;           #Error{:path [:human :address 0 :country]
;                  :in [:address :country]
;                  :schema keyword?
;                  :value "sweden"})}
#2020-07-2915:58ikitommi(Schemas with named branches, e.g. :map and :multi use the name instead of vector index.#2020-07-2923:12danielglauserHow can you validate a "naked" series of integers? Like 1,4,8,10#2020-07-2923:13danielglauserNot in a vector, they are coming in as a query parameter.#2020-07-3007:07ikitommias one string? Maybe something like:
(m/decode 
  [:vector {:decode/string (partial str/split ",")} int?] 
  "1,2,4,8,10"
  mt/string-transformer)
not near a computer, so not 100% sure it works
#2020-07-3007:08ikitommifirst splits the string, the decodes parts string->int#2020-07-3016:07danielglauserThanks, I'll give that a try. I came up with this but your solution seems simpler:
(defn ids?                                                                      
  "Takes in a string and returns true if that string is of the form             
  1,4,7,9 or a sequence (in the math sense) of positive integers."              
  [data]                                                                        
  (let [ids (-> data                                                            
                (clojure.string/split #",")                                     
                (as-> ids (mapv #(Integer/parseInt %) ids)))]                   
    (every? pos? ids)))

(def GetClassesQueryOptions                                                     
  [:map                                                                        
   [:approved {:optional true} boolean?]                                        
   [:instructors {:optional true} [:fn (fn [ids] (ids? ids))]]                  
   [:machine {:optional true} [:fn (fn [ids] (ids? ids))]]                      
   [:music {:optional true} [:fn (fn [ids] (ids? ids))]]                        
   [:bookmarked {:optional true} boolean?]                                      
   [:minlength {:optional true} int?]                                           
   [:maxlength {:optional true} pos?]])
#2020-07-3017:20ikitommiIt's good to keep decoding and validation in separate steps. @alexmiller has many times called the spec conform "a meat grinder", as it bundles the two and runs the transformations every time.#2020-07-3116:32ikitommiAfter #238, there will be two ways to program with schemas: via schema keys (`:in` in explain) and value keys (`:path`in explain). e.g. one can walk easily over parts of schemas that don’t effect the value paths:
(def Schema
  (m/schema
    [:maybe
     [:and
      [:map
       [:id string?]
       [:tags [:set keyword?]]
       [:address
        [:and
         [:map
          [:street {:optional true} string?]
          [:lonlat {:optional true} [:tuple double? double?]]]
         [:fn '(fn [{:keys [street lonlat]}] (or street lonlat))]]]]
      [:fn '(fn [{:keys [id tags]}] (and id tags))]]]))

(mu/get-in Schema [0 0 :address 0 :lonlat])
; => [:tuple double? double?]

(mu/get-in* Schema [:address :lonlat])
; => [:tuple double? double?]
#2020-08-0216:38ikitommiOh my. Accumulating both :in and :path in explain errors seems redundant. If we know Schema and either of schema-path (`:path`) or a value-path (`:in`), we can calculate the other. Simplifies both -explain and the malli utils. Sweet.#2020-08-0216:39ikitommiand the code is dead simple:
(defn in->path [schema in]
  (loop [i 0, s schema, acc []]
    (or (and (>= i (count in)) acc)
        (recur (inc i) (get s (in i)) (if-not (m/-key s) (conj acc (in i)) acc)))))

(defn path->in [schema path]
  (loop [i 0, s schema, acc []]
    (or (and (>= i (count path)) acc)
        (let [[i k] (if-let [k (m/-key s)] [i k] [(inc i) (path i)])]
          (recur i (get s k) (conj acc k))))))
#2020-08-0216:40ikitommi(also, performant, as it’s only protocols and vector indexes)#2020-08-0219:18borkdude@ikitommi Published dynaload to Clojars: https://clojars.org/borkdude/dynaload#2020-08-0308:00ikitommithanks! Will update the deps#2020-08-0411:35ikitommimerged #238 in master. The Breaking: • :path in explain is re-implemented: map keys by value, others by child index • m/-walk and `m/Walker` uses `:path`, not `:in` • m/-outer has new parameter order: `walker schema path children options` • malli.util/path-schemas replaced with `malli.util/subschemas` & `malli.util/distict-by` • LensSchema has a new `-keep` method • renamed some non-user apis in `malli.core` & `malli.util` • moved map-syntax helpers from `malli.core` to `malli.util` • dynaload `com.gfredericks/test.chuck`#2020-08-0411:38ikitommiThere are few more PRs to go, many half-way complete and should unroll after the 238. But, going back to work within few days, don’t know when will have time to finish all the things.#2020-08-0411:46ikitommiApplied for Malli for the Clojurists Together aug-sep, but haven’t heard anything, so just:crossed_fingers:.#2020-08-0510:50ikitommiComments on https://github.com/metosin/malli/pull/194? e.g. allowing spec2-like schema+select using just :map.#2020-08-0510:51ikitommi#2020-08-0514:50teodorluHey! I just found :map-of through after some searching, first reading the Readme, then guessing its name after rechecking the documentation for Spec. Is there interest for a PR adding an example to the Readme?#2020-08-0515:56ikitommidefinetely! All doc improvements most welcome#2020-08-0515:57ikitommisomeone suggested setting up real docs too, under docs so that cljdoc would pick them up too#2020-08-0614:39ikitommiwith current master:#2020-08-0614:39ikitommi#2020-08-0708:46borkdudeHoe does one combine validate and transform? E.g. this doesn't crash:
(prn (m/decode int? :foo mt/string-transformer))
I'm not saying it should, just wondering how to do it. Not clear from the README
#2020-08-0708:48borkdudeShould I first call m/valid, if not valid, then m/explain and else m/decode, effectively traversing the structure twice?#2020-08-0709:21ikitommiIf you need decoding, the flow should be: 1. decode 2. validate 3. explain on error#2020-08-0709:22ikitommithere is 2-3 walks in the error case.#2020-08-0709:22ikitommifor happy case, 1-2#2020-08-0709:24ikitommihaving a seoarate optimized validate makes the happy case fast#2020-08-0709:24ikitommithe docs could have examples on this...#2020-08-0709:26ikitommithe m/decoder doesn't have to walk the structure, it returns an function to transform just the parts that need to be decoded. In case there is nothing to do, it returns identity#2020-08-0709:35borkdude@ikitommi The concrete example I was going to try:
$ cat deps.edn
{:deps {metosin/malli {:git/url "" :sha "2bd749f7148e28a379f1e628a32188e7f6cf0bc4"}
        borkdude/edamame {:git/url "" :sha "64c7eb43950eb500ba7429dded48257cd15355ae"}}}
$ cat src/edamalli/core.clj
(ns edamalli.core
  (:require [edamame.core :as e]
            [malli.core :as m]
            [malli.transform  :as mt]))

(defrecord WrappedNum [obj loc])

(defn postprocess [{:keys [:obj :loc]}]
  (if (number? obj) (->WrappedNum obj loc) obj))

(defn -main [& args]
  (prn (e/parse-string "[:foo 42]" {:postprocess postprocess}))
  ;; TODO:
  ;; - validate that the WrappedNum contains value < 42
  ;; - then transform it to only that number
  ;; - else raise error, printing the location metadata of that number
  )
$ clojure -m edamalli.core
[:foo #edamalli.core.WrappedNum{:obj 42, :loc {:row 1, :col 7, :end-row 1, :end-col 9}}]
#2020-08-0711:06zclj@ikitommi I am parsing a schema into a malli-schema. The original might contain recursive references to other "entities" in the schema. I do not know this up front. Are there any trade-offs in putting all my potential recursive entity references in a [:ref ], even if they turn out not to be?#2020-08-0711:44ikitommi@borkdude would [:foo 42] be transformed to [:foo 42] , as would [:foo :bar #{42}] to itself and {:a 41} would fail on the fact that there was a number that was not 42?#2020-08-0711:45ikitommihappy to help, sample inputs -> outputs would help.#2020-08-0711:45borkdude@ikitommi No, [:foo (WrappedNum. x y)] would be transformed to [:foo x] only if x < 42, else error with explain using y#2020-08-0711:46ikitommicould that validation happen already in the :postprocess?#2020-08-0711:47borkdudeI just want to feed this data to malli and not intertwine parsing data from text to sexprs with malli validation#2020-08-0711:48ikitommiok. can the input be anything, e.g [:foo {:bar (WrappedNum. x y)}]?#2020-08-0711:48borkdude[:foo y] could also be {:foo y} if that's easier for you#2020-08-0711:49borkdudeNo, it's more like person: {:name "foo" :age 42}, let's say#2020-08-0711:49borkdudeso attribute+value#2020-08-0711:50borkdudeso unwrapping would just be (:obj wrapped), that would be the transform step#2020-08-0711:50borkdudeThe use case for this is: normally edamame doesn't let you have location metadata for numbers and strings, but using a wrapped value you can have that#2020-08-0711:51borkdudeso I want to use malli like normally, but just use the location metadata in the wrapped value for reporting errors and discard it if the value is valid#2020-08-0711:59borkdudeI believe in spec you would write a conformer for this#2020-08-0712:12ikitommiyes, this is kinda tricky with current malli, as there is not yet a parsing api, like conform.#2020-08-0712:13ikitommialso, there is no custom overridable validator, so one needs to describe the given data structure (here, a tuple with keyword and a record).#2020-08-0712:15borkdudeok, so one would write a schema using the records and if everything's ok, then postwalk yourself, unwrapping them?#2020-08-0712:15ikitommibut, something like this:
;; schemas
(def <42 [:and int? [:< 42]])
(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure :loc} [:obj <42]]])

;; validator and encoders for both success & failure
(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))

;; in action
(defn parse-validate-and-transform [s]
  (let [x (e/parse-string s {:postprocess postprocess})]
    (if (valid? x) (success x) (failure x))))
=>
(parse-validate-and-transform "[:foo 41]")
; => [:foo 41]

(parse-validate-and-transform "[:foo 42]")
; => [:foo {:row 1, :col 7, :end-row 1, :end-col 9}]
#2020-08-0712:16ikitommiyes, postwalk would do. or a recursive schema definition. if the wrapped records can be anywhere#2020-08-0712:16borkdudelet me try your snippet#2020-08-0712:17borkdude
$ clojure -m edamalli.core
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:76).
:sci-not-available {:code ":obj"}
That was unexpected, I don't need sci in this example?
#2020-08-0712:18ikitommioh, should not.#2020-08-0712:19ikitommi
(def Schema [:tuple keyword? [:map {:encode/success (fn [x] (:obj x)), :encode/failure (fn [x] (:loc x)) [:obj <42]}]])
#2020-08-0712:21ikitommipushed e19872273c3660fbc482dcff4c2d8439dbcbb2a6, which should allow naked keywords as functions.#2020-08-0712:21ikitommi
(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure :loc} [:obj <42]]])
#2020-08-0712:21borkdude#2020-08-0712:22borkdudeI'm still getting the sci-not-available error#2020-08-0712:23borkdudeWhen I do include sci, I get:
[:foo {:row 1, :col 7, :end-row 1, :end-col 9}]
#2020-08-0712:24borkdudeI guess I should throw my own error in :encode/failure?#2020-08-0712:26ikitommiyes, that’s one place to do it. will check why sci is needed. just a sec.#2020-08-0712:27borkdude
(defn parse-validate-and-transform [s]
  (let [x (e/parse-string s {:postprocess postprocess})]
    (if (valid? x)
      (prn "SUCCESS" (success x))
      (prn "ERROR" (failure x)))))
$ clojure -m edamalli.core
"ERROR" [:foo {:row 1, :col 7, :end-row 1, :end-col 9}]
#2020-08-0712:29borkdudehaha, when I do this:
:encode/failure {:message "should be lower than 42"}
I get:
$ clojure -m edamalli.core
Execution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:54).
Could not resolve symbol: should [at line 1, column 1]
#2020-08-0712:29borkdudeI have no idea what I'm doing, since I don't know these APIs well. I'll take a look after work again some time#2020-08-0712:36ikitommi8e067b3d004d1692cbfc695bc73d7e032ecb6e7f#2020-08-0712:37ikitommithe code used sci for all non fn?s, no to all non ifn?s. need to add tests.#2020-08-0712:38borkdude@ikitommi I now have this:
(def Schema [:tuple keyword? [:map {:encode/success :obj, :encode/failure {:message "should be lower than 42"}} [:obj <42]]])
Output:
"eval!" "should be lower than 42"
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:76).
:sci-not-available {:code "should be lower than 42"}
#2020-08-0712:39borkdudeI might be doing something wrong, but it seems there's a debug println in there?#2020-08-0712:43ikitommipicard-facepalm my bad. but this kinda works (but is bad, should be better when we have the parsing api):#2020-08-0712:43ikitommi
➜  ~ clojure -Sdeps '{:deps {metosin/malli {:sha "230b1767729aad3e02568f1320855e2b45d2d9b5", :git/url ""}, borkdude/edamame {:sha "64c7eb43950eb500ba7429dded48257cd15355ae", :git/url ""}}}'
Checking out:  at 230b1767729aad3e02568f1320855e2b45d2d9b5

Clojure 1.10.1
user=> (ns edamalli.core
  (:require [edamame.core :as e]
            [malli.core :as m]
            [malli.transform :as mt]))

(defrecord WrappedNum [obj loc])

(defn postprocess [{:keys [:obj :loc]}]
  (if (number? obj) (->WrappedNum obj loc) obj))

(defn fail! [{:keys [:obj :loc]}]
  (throw (ex-info (str "so bad " obj "/" loc) {})))

(def <42 [:and int? [:< 42]])
(def Schema [:tuple keyword? [:map {:encode/success :obj,
                                    :encode/failure fail!} [:obj <42]]])

(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))

(defn parse-validate-and-transform [s]
  (let [x (e/parse-string s {:postprocess postprocess})]
    (if (valid? x) (success x) (failure x))))

edamalli.core=> (parse-validate-and-transform "[:foo 41]")
[:foo 41]

edamalli.core=> (parse-validate-and-transform "[:foo 42]")
Execution error (ExceptionInfo) at edamalli.core/fail! (REPL:2).
so bad 42/{:row 1, :col 7, :end-row 1, :end-col 9}
#2020-08-0712:45ikitommi@zclj there is a small (have not measured) penalty for using ref-schemas, one function hop basically as the values are memoized.#2020-08-0713:03zcljok, that's fine since I have to do something to solve it anyway, by post-walking or such. Doing it up-front with malli ref considerable make the design simpler. Thanks for the info!#2020-08-0717:50ikitommirollback on the perf info @zclj . Validation & explain perf is about the same but transforming values is potentially much slower. Why? Malli can't optimize over :refs. Transformation behind :ref could be no-op, but it won't be removed as it's wrapped in a function. Still, orders of magnitude faster than with spec(-tools)#2020-08-0809:45zcljthanks for the update! In my use-case I will also do generation from the schema, where I will blow the stack if I don't use :ref for recursive references. Are there any implications for using :ref for potentially non-recursive entities in that case?#2020-08-0712:45borkdudeworks, thanks#2020-08-0717:50ikitommirollback on the perf info @zclj . Validation & explain perf is about the same but transforming values is potentially much slower. Why? Malli can't optimize over :refs. Transformation behind :ref could be no-op, but it won't be removed as it's wrapped in a function. Still, orders of magnitude faster than with spec(-tools)#2020-08-1012:02Shuai LinHow can I express a schema of "map of string keys to integer values?" in malli?#2020-08-1012:03ikitommi@linshuai2012 [:map-of string? int?]#2020-08-1012:03Shuai LinThanks @ikitommi, just found it's there in README, I read through the README and missed it 😄#2020-08-1012:08Shuai Linbtw thanks for creating malli, it's much data friendly and intuitive than spec ! I'm most looking forward to see the fdef equivalent https://github.com/metosin/malli/issues/125#2020-08-1012:08ikitommime too 😉#2020-08-1012:19borkdudeI'm also looking forward to a s/conformer equivalent#2020-08-1013:24zclj@ikitommi I have a schema where some maps have a lot of optional keys, up to 50-ish. When I generate from this schema I got a, to me, surprising result. What happens is that even for very small sizes I get very large generated maps (and since they are recursive this is magnified). Looking into how malli generates maps with optional keys, it seems that each key will have a 50% change of being included, so there is no notion of 'size' for the complete map. This also effect shrinking. So in practice my first generated map can be empty, while the second sample contains 20 keys. Intuitively I would expect the generation of optional keys to start small and then grow larger with size. Is this something you think malli should take care of or is it out of scope for malli and something I need to take care of with custom generators?#2020-08-1013:28ikitommi@zclj if you have a simple improved algorithm for generating with optional keys, please PR, the code is kinda naive atm: https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L84-L87#2020-08-1308:11zclj@ikitommi I have tried out some different solutions to this and though about it. My conclusion so far is that there are really no perfect default solution. As is usally the case in software testing (where I use the generators) the proper distribution of values are dependent on both the system under test and the outcome you want (positive tests, negative tests etc.) With this in mind, would you be open to a solution where malli could allow for the options to contain a :malli.generators/map-gen-optional-fn, where the user can provide their own implementaiton of the value selection and keep the default as is? In general, if malli provided such options it would be an awesome way to experiment with different distributions and allow for libraries to provide commonly used variants.#2020-08-1308:15ikitommiwith the current api, you can override the whole :map generator with:
(defmethod malli.generator/-schema-generator :map [schema options] (-my-map-gen schema options))
#2020-08-1308:16ikitommiwould that be enough?#2020-08-1308:17ikitommi(not happy that one can globally override the generators, I concider this as mutable evil, but is like that today)#2020-08-1308:29zcljI experimented some with that option but did not pursue it further due to me still wanting the -recur funtionallity and then having to do that in my own code. I could of course re-use mallis -recur but it felt that I started to depend to much on malli internals for this to be a clean method of extension, but maybe I was to defensive in that decision?#2020-08-1308:41ikitommineed to publish a malli internal naming convention guide. my idea was that all public - starting functions are public for library extensions / advanced use, tracked separately in CHANGELOG from the public end user api. Using those for basic use cases is a sign of invalid usage. Those will break more than the user-facing public api, but so that it’s easy for advanced users adapt to changes, e.g. no silent breaking changes after initial public release.#2020-08-1308:42ikitommibut, on second thought, if that one new option is good / you can provide alternative example impl (into README/docs) that might be of value to someone else, open to PR for adding that.
#2020-08-1308:43ikitommias options are passed in to -map-gen already, it’s most likely <4 lines of code + tests + docs.#2020-08-1308:43ikitommi(and doesn’t add much to cljs bundle size)#2020-08-1413:27eskosIs there a reason the private stuff simply isn’t defn- ? Are the private parts split across multiple namespaces in such a way it would make this complicated?#2020-08-1413:51ikitommireal privates are ^:private. Vars like malli.core/-map-schema are public but only needed if you build your own registry, which is for advanced users only. Another is malli.core/-parse-entries which is a helper for building your own Schema instance, which wants to use the map-syntax#2020-08-1413:52ikitommispec is mostly closed for extensions, malli is built to be extended.#2020-08-1013:32zclj@ikitommi I will take a look and see if I can figure something out 🙂#2020-08-1014:01Shuai Linanother question, can I add some meta information to each field? Like this:
(def Person
  [:map
   [:name string? {:doc "The name of the person"}]
   [:age string?]])
#2020-08-1014:11ikitommi@linshuai2012 map entries can have optional property map:
(def Person
  [:map
   [:name {:doc "The name of the person"} string?]
   [:age string?]])
#2020-08-1014:12ikitommijust polishing the generic traversal of child-parents:
(defn parent-properties [schema path]
  (loop [i (count path), acc []]
    (if (>= i 0)
      (let [p (subvec path 0 i)
            s (mu/get-in schema p)
            ++ #(conj % (m/properties s))]
        (if (and (m/entries s) (< i (count path)))
          (recur (dec i) (++ (conj acc (m/properties (mu/get-in schema (conj p [::m/entry (path i)]))))))
          (recur (dec i) (++ acc))))
      acc)))

(parent-properties
  [:map {:error/message "1"}
   [:y {:error/message "2"}
    [:and {:error/message "3"}
     [:map {:error/message "4"}
      [:x {:error/message "5"}
       [:and {:error/message "6"}
        int? [:> {:error/message "7"} 18]]]]]]]
  [:y 0 :x 1])
;[#:error{:message "7"}
; #:error{:message "6"}
; #:error{:message "5"}
; #:error{:message "4"}
; #:error{:message "3"}
; #:error{:message "2"}
; #:error{:message "1"}]
#2020-08-1016:19ikitommi@borkdude collected some thoughts on the s/conform here: https://github.com/metosin/malli/issues/241. Comments and ideas welcome.#2020-08-1017:00borkdudeCool! I was asking about s/conforming specifically which was a spelling error, it should be https://clojuredocs.org/clojure.spec.alpha/conformer#2020-08-1017:03borkdudeThis was the "transform while conforming" issue#2020-08-1308:41ikitommineed to publish a malli internal naming convention guide. my idea was that all public - starting functions are public for library extensions / advanced use, tracked separately in CHANGELOG from the public end user api. Using those for basic use cases is a sign of invalid usage. Those will break more than the user-facing public api, but so that it’s easy for advanced users adapt to changes, e.g. no silent breaking changes after initial public release.#2020-08-1114:35borkdude@ikitommi do the names :encode/conform and (mt/transformer {:name :conform}) match? why is mt/transform not called mt/encode?#2020-08-1114:36borkdudeah because it also has decode, gotcha#2020-08-1207:08ikitommithe transformation terms: • transformation function, a function of A->B, e.g. converts strings to dates • decoding, a process of transforming (invalid) values into potentially valid ones (IN*), m/decoder& m/decode* • encoding, a process of transforming (valid) values into something else (OUT), m/encoder & m/encode • transformer, a top-level component that maps Schemas with transformation functions (“json-transformer transforms strings to dates, but not strings to numbers”). Needed in encoding and decoding, mt/transformer • named transformer, If a transformer has :name defined, Schemas can define their transformation functions (for both encoding & decoding) using Schema properties • interceptor, a component that bundles transforming functions into transforming phases • transforming phase, either :enter or :leave, timing when a transformation function is applied in the chain (before the fact or after the fact) • interceptor chain, a sequence of interceptors that is used to run the (optimized chain of) transformation functions from interceptors in correct order • transformation chain , transformers compose too: (mt/transformer {:name :before} mt/json-transfomer {:name :after})#2020-08-1210:33eskosCould be valuable to add these as glossary doc to the repository (`docs/glossary.md`) to document them 👍#2020-08-1207:08ikitommilot of things inside to make things composable and fast, the basic user doesn’t have to know much of the details.#2020-08-1207:23ikitommi
(m/decode
  [:map
   [:x {:default 42} int?]
   [:y [:vector {:decode/before #(str/split % #",")
                 :decode/after {:leave #(mapv inc %)}} int?]]]
  {:y "1,2,3"}
  (mt/transformer
    {:name :before}
    (mt/default-value-transformer)
    (mt/string-transformer)
    {:name :after}))
;{:y [2 3 4]
; :x 42}
#2020-08-1207:24ikitommi
(m/decode
  [:map
   [:x {:default 42} int?]
   [:y [:vector {:decode/before #(str/split % #",")
                 :decode/after {:leave #(mapv inc %)}} int?]]]
  {:y "1,2,3"}
  (mt/transformer
    {:name :before}
    (mt/default-value-transformer)
    (mt/string-transformer)
    {:decoders {'int? #(* % 2)}}
    {:name :after}))
;{:y [3 5 7]
; :x 84}
#2020-08-1709:15Shuai Linis it possible for my transformer/decode to know the current path when decoding? e.g. in the below example, I'd like to replace the <location> with :k1
(ma/decode
 [:map 
  [:k1 [:map {:decode/foo #(assoc % :location :<location?>)}
        [:x int?]]]]
 {:k1 {:x 1}}
 (mt/transformer
  {:name :foo}))
;; => {:k1 {:x 1, :location :<location?>}}
#2020-08-1709:17Shuai Linthe background story is I want to generate a unique key for each such map based on its path in the whole tree#2020-08-1709:45ikitommi@linshuai2012 not at the moment, but you can do the following: 1) use m/walk to transform the schemas by adding a :in property to schemas (Schema :path and value :in are available for walkers) 2) create a decoder that uses the interceptor :compile hook to access the Schema at decoder creation time#2020-08-1709:46ikitommithere are few pending PRs around walkers, not 100% sure what is in the current master.#2020-08-1709:48Shuai Lin@ikitommi thx! I'll take a look at this approach. Currently I'm trying a solution that 1. leaves a place holder when decoding, and then 2. after decoding the whole tree, walk my tree on my own (and accumulate the path), and for each such map, use clojure.walk/prewalk to replace the placeholder with the current path#2020-08-1709:50ikitommipretty sure it's not very performant that way. There is an example of attaching a generated sample value to all schemas in the readme.#2020-08-1709:51ikitommiIt uses the m/schema-walker , you need the plain walker to access the :in data. But otherwise, the 1) should be copyable from that.#2020-08-1712:03Shuai LinJust realized it's not what I want, the schema walker walks the schema, not the actual data. I still need to walk the actual data to get the pat to generate the unique key for each map in the tree based on its actual location. Also, my schema is recursive - think about a file system, where there are directories as maps, files as leaves, and sub-directories as child maps.#2020-08-1712:40ikitommiwalk + decode would work for non-recursive schemas. Please write an issue, I'll think about the solution.#2020-08-1715:26Shuai LinThx! I will.#2020-08-1709:52ikitommialso, the interceptor :compile hook could publish more information about the context to the transformers, last call to break the api before freezing things. Please write an issue if you want that#2020-08-1709:53ikitommiIt is now a callback fn with args of schema value, could be schema value path in root-schema for example#2020-08-1709:57Shuai Linthe m/walker apporach looks promising, I'll try it next#2020-08-1710:18CaseyI'm just getting started exploring malli#2020-08-1710:19Caseyhttps://github.com/metosin/malli#mallicorebase-schemas seem to indicate if I want a :int or :int-in schema, I'd have to register it myself, is that correct?#2020-08-1710:24ikitommi@ramblurr you can just create a function that returns a form for it:
(defn int-in [min max]
 [:and int? [:fn '(fn [x] (< min x max))]])
#2020-08-1710:25ikitommibut you should add a :gen/gen for it too. I think there should be a built-in for that (and for dates too)#2020-08-1710:26ikitommi
[:int {:min 1, :max 10}]
[:date {:min "2020-08-16", :max "2020-10-10"}]
kinda things
#2020-08-1710:28ikitommithe raw impl would look much like :string: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L772-L805#2020-08-1710:28Caseyah great, i see#2020-08-1710:28Caseyis there an aggregated list of built ins somewhere (with docs as to their properties)#2020-08-1710:29Casey> I think there should be a built-in for that (and for dates too) Do you mean you think this built-in already exists, or you had the idea that it should be added ? 🙂#2020-08-1710:29ikitommi(m/default-schemas) gives a list of all. the source code is currently the best source of descriptions.#2020-08-1710:29ikitommishould be added 🙂#2020-08-1710:30ikitommithe properties - will add description of those using malli (eat your own..), but not there yet.#2020-08-1710:33CaseyUsing the int-in function def you gave an example for above.. if you added it to a registry {:int-in (m/fn-schema :int-in int-in)} , how could it be consumed later? In fact, this wouldn't work right?#2020-08-1710:35CaseyYou could do something like {:int-in-0-10 (m/fn-schema :int-in-0-10 (partial int-in 0 10))} I suppose#2020-08-1710:44ikitommi
(defn int-in [min max]
  [:and int? [:fn `(fn [x] (< ~min x ~max))]])

(def int-in-1-10 (int-in 1 10))

(validate [:tuple int-in-1-10 (int-in 10 100)] [2 12])
; => true

(form [:tuple int-in-1-10 (int-in 10 100)])
;[:tuple
; [:and int? [:fn (clojure.core/fn [malli.core/x] (clojure.core/< 1 malli.core/x 10))]]
; [:and int? [:fn (clojure.core/fn [malli.core/x] (clojure.core/< 10 malli.core/x 100))]]]
#2020-08-1710:46ikitommiif you write an issue about the :int as built-in, happy to add that. much cleaner#2020-08-1711:15CaseyAm I correct that you can't use the generic :int-in as defined above in a registry, because the "registered" specs must be predicate predicate functions (single value as input)?#2020-08-1711:24ikitommi• registered schemas are one of: 1) IntoSchema instance (e.g. something that take the schema syntax and return a Schema. 2) Schema instance, 3) Schema syntax. • m/fn-schema just takes a predicate fn, which is only used to build a validator for the schema • there could be more helpers to build custom schemas easier, something between m/fn-schema (here: too simple) and writing IntoSchema impl (here: too much work) by hand, but currently, there is not.#2020-08-1711:26ikitommiI think I’ll extract the code from :string so that :date, :number, :date-time etc can reuse most of it (e.g. the :min + :max handling of via properties, effecting both validation and value generation)#2020-08-1711:28ikitommialso, you could build your own things with it easily: {:registry {:bigdec (m/-ranged-pred-schema {:pred bigdec?, :range-pred …, :range-gen ...})}}#2020-08-1712:03Shuai LinJust realized it's not what I want, the schema walker walks the schema, not the actual data. I still need to walk the actual data to get the pat to generate the unique key for each map in the tree based on its actual location. Also, my schema is recursive - think about a file system, where there are directories as maps, files as leaves, and sub-directories as child maps.#2020-08-1714:35Casey@steveb8n I've been playing with your demo (https://github.com/stevebuik/fork-malli-ideas) , it's a really nice approach to form validation.#2020-08-1722:14steveb8n@UE35Y835W nice to see some interest#2020-08-1907:32LuYeah pretty cool!! 😎 #2020-08-1912:44Luin 2.1.4 you can add a keywordize-keys true option to work exclusivley with keywords/namespaced keywords instead of strings, also in the validation#2020-08-1714:36CaseyThe keyword <-> string transformations are clever too, though don't work when namespaced keys are used#2020-08-1715:40CaseyI'm working around this using (defn unkeyword [m] (map-keys #(subs (str %) 1) m)) at the end of validator-for-humans to turn :foo/attr-> "foo/attr" .. and of course using "foo/attr" as the name in the form form. Works well enough, if a little dirty.#2020-08-1714:37ikitommi@ramblurr https://github.com/metosin/malli/pull/243/commits/7d335e916d9769d4039b69e1475f04f3239ac5c7#2020-08-1714:38ikitommimerged in master#2020-08-1714:39Caseyfirst look comment: the max range should be exclusive, not inclusive. That's a pretty standard trope across all range checks in almost any language I know#2020-08-1714:39ikitommiwith m/-simple-schema it should be easy to add new schemas that use properties in validation:
(-simple-schema {:type :double, :pred double?, :property-pred (-min-max-pred identity)}))
#2020-08-1714:40ikitommi:thinking_face:#2020-08-1714:40ikitommispec has that, test.check uses inclusive for both.#2020-08-1714:40ikitommicould you link some external wisdom for that?#2020-08-1714:40Caseyhttps://clojuredocs.org/clojure.spec.alpha/int-in-range_q is exclusive#2020-08-1714:41ikitommihttps://clojure.github.io/test.check/clojure.test.check.generators.html#var-large-integer* is inclusive#2020-08-1714:42ikitommiJSON Schema has inclusive: https://json-schema.org/understanding-json-schema/reference/numeric.html#2020-08-1714:43Caseyand so s/int-in and s/double-in are exclusive, https://clojuredocs.org/clojure.core/range is exclusive,#2020-08-1714:43Caseyjson schema supports inclusive and exclusive options 😛#2020-08-1714:46Caseyto be clear, i'm advocating only the max be exclusive, look at pretty much and langugage: python's list slice operator, java's IntStreams,#2020-08-1714:47Caseyhere's an argument why: https://wrschneider.github.io/2014/01/07/time-intervals-and-other-ranges-should.html (starting off about time, but then at the end mentions integer ranges)#2020-08-1714:48CaseyHaha, and https://stackoverflow.com/questions/8441749/representing-intervals-or-ranges is a link to https://www.cs.utexas.edu/users/EWD/transcriptions/EWD08xx/EWD831.html by Dijkstra on the topic#2020-08-1714:48ikitommithanks. will read those.#2020-08-1714:50CaseyThat said, I don't need to die on this hill 🙂 Just sharing my experience that generally when I see a range in an api, I assume (and assumed others did too!) that it was inclusive start and exclusive end. As long as it's documented, it'll be ok either way#2020-08-1717:25ikitommimeanwhile:
(mg/generate
  [:map
   [:string :string]
   [:int :int]
   [:double :double]
   [:boolean :boolean]
   [:keyword :keyword]
   [:symbol :symbol]
   [:qualified-keyword :qualified-keyword]
   [:qualified-symbol :qualified-symbol]]
  {:size 42, :seed 42})
;{:string "¦®GÏVá@£°5o,&µ7\rØã",
; :int -1251,
; :double -0.03125,
; :boolean true,
; :keyword :WD_VS_-r,
; :symbol k5K_2i,
; :qualified-keyword :M8qL/u?RAmf,
; :qualified-symbol x/y0T}
related: https://github.com/metosin/malli/issues/25
#2020-08-1912:15CaseyGiven a registry map, is there a function that will return a list of the qualified keys in the registry?#2020-08-1914:55ikitommino, there isn't#2020-08-2006:07ikitommi... but just filter or reduce the keys?#2020-08-2007:00CaseyYup, that's what I did 🙂#2020-08-2007:03CaseyI'm finding malli very useful for client and server side entity validation. When you have an entity that a client can submit, but is only allowed to submit a subset of all possible attributes and the other attributes are populated by the server. In a ns for an entity, I have one main custom registry (which is just a map!) that defines all possible attributes of the entity, then I build little schemas that pick out the "views" of what a valid entity looks like at different stages in its lifecycle.#2020-08-2011:45CaseyWhat's the story behind the name malli, is there one?#2020-08-2012:28ikitommihttps://www.slideshare.net/metosin/malli-inside-datadriven-schemas#21 (slide 21)#2020-08-2113:07CaseyFollowing the local registry example at https://github.com/metosin/malli#local-registry calling (malli.core/schema? Adult) => false . This surprises me..#2020-08-2113:09CaseyIs it not a schema? Why not?#2020-08-2113:12ikitommiit’s just data?#2020-08-2113:23CaseyAh, I see. I see it expects a Schema protocol.... and you can use m/schema to turn the data into a Schema entity#2020-08-2113:24ikitommiI think the m/schema? is not that useful, could be removed.#2020-08-2113:25ikitommias most of the functions consume data, Schema or IntoSchema .#2020-08-2113:25Caseym/schema? is used in some fdef specs in gungnir#2020-08-2512:09Elso
"No implementation of method: :-form of protocol: #'malli.core/Schema found for class: clojure.core$int_QMARK_"
Version bumped from "0.0.1-20200404.091302-14" to "0.0.1-SNAPSHOT". Validating with
(->> (m/decode schema data mt/string-transformer)
     (m/explain schema)
     me/humanize))
and a schema like
[:map
  [:something [:map
                [:a int?]]]
and input data like
{:something {:a 10}}
#2020-08-2512:09ElsoWhat's going on here?#2020-08-2513:19ElsoApparently, my example was not telling the important part and the breaking change was that this: [:map [:issue keyword? :user-endpoint string?]] used to work but is now required to be [:map [:issue keyword?] [:user-endpoint string?]]#2020-08-2513:19ElsoWhich seems quite reasonable#2020-08-2515:56ikitommiplan is to describe malli schema syntax using malli to get humanized errors on invalid syntax#2020-08-2906:39ikitommi🎉#2020-08-3011:17borkdudeCongrats on the CT grant :)#2020-08-3011:17borkdudeWill malli have something like spec/fdef + instrument + unstrument? And will the sequential destructuring be part of the first production release?#2020-08-3015:51ikitommi@borkdude the first release is planned to be just the current and (hopefully) immutable core on which everything builds on, Here’s the roadmap for the together funding:
1) get a stable release out! lot's of small and some bigger design decisions, tracked via metosin/malli#116

2) help early adopters (users and libraries like reitit, regal, aave and gungnir) to upgrade to use the initial version

After the release, would work on the following:

3) finalize sequence schemas, 

4) enhance developer tooling: 
  - function schemas with clj-kondo integration 
  - pull out and reuse the reitit development time error pretty printer as a separate library ()

5) implement pluggable schema inference

6) parsers
#2020-08-3015:53ikitommidestucturing might be easy to implement using the current explain api, need to visit that for the first release to see will it require changes to the m/explain format.#2020-08-3015:54ikitommiVincent has done a great initial work already with the sequence schemas, will dig into that soon.#2020-09-0119:03Vincent CantinThx ^_^ I recommend to take the impl from Minimallist as a reference, as it is the most up-to-date and (most importantly) well tested. The work remaining to be done is integration and optimization.#2020-09-0209:35ikitommiwas just about to ask from which codebase should I look this for. Thanks!#2020-09-0210:48Vincent Cantinthe latest commit of the main branch all-work-and-no-play#2020-08-3015:54ikitommiso, no, but hopefully soon has both.#2020-08-3015:54borkdude:thumbsup:#2020-08-3015:59ikitommimerged https://github.com/metosin/malli/pull/249. Adds :double,  `:boolean`,  `:keyword`,  `:symbol`,  `:qualified-keyword`, `:qualified-symbol` and `:uuid`. There is now m/-simple-schema to easily build custom schemas using properties in validation. Could rewrite m/-predicate-schema to use that too.#2020-08-3106:08ikitommihttps://github.com/metosin/malli/pull/212 would remove m/map-entries in favor of m/entries retuning a sequence of clojure.lang.MapEntry. m/children is quaranteed to return the tupl-3 of [key properties schema] in the future. This is a breaking change, so comments welcome.#2020-08-3106:10ikitommirelates to 5 issues and 2 PRs, one of the last design decisions that needed re-visiting.#2020-09-0120:55shaunxcodeHave recursive schemas been implemented now? I know there was a ticket (which I cant find right now) where various versions were being discussed.#2020-09-0121:07jhacks@shaunxcode It looks like this might be the discussion: https://github.com/metosin/malli/pull/117 and maybe this is the implementation?: https://github.com/metosin/malli/pull/209 but I’m just getting familiar with malli#2020-09-0121:10shaunxcodethanks that is it exactly!#2020-09-0121:14jhacks@shaunxcode I just saw this section in the readme: https://github.com/metosin/malli#recursive-schemas which might be helpful too#2020-09-0121:39jhacksI was following along with this code from the README.md:
;; regexs work too
(mg/generate 
  [:re #"^[a-zA-Z0-9._%+-]
In order for it to work I needed to explicitly add test.chuck as a dependency, or else I would get this error:
Execution error (IllegalStateException) at malli.generator/eval20132$fn (generator.cljc:135).
    Attempting to call unbound fn: #'malli.generator/-re-gen
If that’s the expected behavior (and I didn’t mess something up), maybe it would be good to add a note to the README.md under https://github.com/metosin/malli#value-generation about adding test.chuck as a dependency?
#2020-09-0204:59ikitommi@jhacks documented the dependency to README and made it fail better: https://github.com/metosin/malli/commit/5c3689fd42f2c73253a0f81a88b23e20a3b6417b#2020-09-0205:02ikitommiI believe regal has regex generators for both clj & cljs, could use that instead. It’s still wip according to README: > The following aspects have known issues or are otherwise untested or incomplete, and you can expect them to change significantly as we further develop them: > * Creating test.check generators from regal forms#2020-09-0205:03ikitommimaybe @plexus could verify the readiness there? (there is also a schema type for malli in regal, which is awesome!)#2020-09-0214:09ElsoIs there something like nillable in malli? Because
(malli.core/validate [:map [:a {:optional true} string?]] {:a nil})
=> false
(malli.core/validate [:map [:a {:optional true} string?]] {})
=> true
this is rather odd to me.
#2020-09-0214:13jeroenvandijkI’m wondering if Malli supports lazy registries. I’m guessing it should be possible with the registry protocol. Anyone know of examples?#2020-09-0214:14jeroenvandijkI want to build an AWS Cloudformation validator, but I don’t want to load all schema files upfront. By doing it lazy I hope to win some startup time#2020-09-0214:31ikitommiSounds like a great idea. Just implement something lazy behind malli.registry/Registry and plug it in.#2020-09-0214:35jeroenvandijkThanks! I’ll give it a try#2020-09-0214:47jeroenvandijkSo the value defines what needs to be looked up. E.g. {:Type "AWS::EC2::Instance" …} would need to be validated with the schema for "AWS::EC2::Instance" , so I need to inspect the value before I know the type. Might make it a bit trickier#2020-09-0214:27jhacks> documented the dependency to README and made it fail better Thanks, @ikitommi! 🙌#2020-09-0214:36ikitommi@d.eltzner012
(m/validate [:maybe :string] nil)
; => true

(m/validate [:maybe :string] "sheep")
; => true
#2020-09-0214:52ElsoThanks a lot!#2020-09-0215:36jhacks> I believe regal has regex generators for both clj & cljs, could use that instead Wow, thanks for mentioning regal, I was not aware of it: https://github.com/lambdaisland/regal#use-with-malli#2020-09-0305:23ikitommi@jeroenvandijk wanted to test the lazy registries.
(require '[malli.core :as m])
(require '[malli.registry :as mr])
Given a data-source that can map names to schemas:
(def schema-provider
  {"int" :int
   "map" [:map [:x "int"]]
   "maps" [:vector "map"]})
We can compose a registry that uses both local and lazy/external resolving:
(defn LazyRegistry [default-registry]
  (let [cache* (atom {})
        registry* (atom nil)]
    (reset!
      registry*
      (mr/composite-registry
        default-registry
        (reify
          mr/Registry
          (-schema [_ name]
            (or (@cache* name)
                (do (println "loading" (pr-str name))
                    (when-let [schema (schema-provider name)]
                      (swap! cache* assoc name (m/schema schema {:registry @registry*}))
                      schema))))
          (-schemas [_] @cache*))))))

(def registry (LazyRegistry m/default-registry))
Using the registry (either swap the m/default-registry or pass as argument:
(count (mr/-schemas registry))
; => 125

(m/validate "map" {:x 1} {:registry registry})
;loading "map"
;loading "int"
; => true

(m/validate "map" {:x 1} {:registry registry}) ;; cached
; => true

(count (mr/-schemas registry))
; => 127

(m/validate "maps" [{:x 1}] {:registry registry})
;loading "maps"
; => true

(count (mr/-schemas registry))
; => 128
Schemas are first class :refs:
(m/schema "map" {:registry registry})
; => "map"

(m/-deref (m/schema "map" {:registry registry}))
; => [:map [:x "int"]]
Hope this helps.
#2020-09-0307:56jeroenvandijk@ikitommi Thanks for sharing. I think it’s almost what I need. I’m puzzling how to deal with the (lazy) dispatch on a map key. In clojure.spec I would use multimethods and multispec:
(defmulti resource-type :Type)

(s/def :aws.cfn/resource (s/multi-spec resource-type :Type))

;; Some random examples
(defmethod resource-type "AWS::AmazonMQ::Broker" [_] :aws.amazon-mq/broker)
(defmethod resource-type "AWS::AmazonMQ::Configuration" [_] :aws.amazon-mq/configuration)
(defmethod resource-type "AWS::ApiGateway::Account" [_] :aws.api-gateway/account)
(defmethod resource-type "AWS::ApiGateway::ApiKey" [_] :aws.api-gateway/api-key)
...
If I can do this dispatch somehow, with your suggestion I think I have all I need
#2020-09-0308:02jeroenvandijkI’ll study the :multi schema and see if that is the missing piece#2020-09-0308:08ikitommis/multi-spec is open & mutable, :multi is closed & immutable.#2020-09-0308:09ikitommiso here, I think a lazy multi variant would be needed.#2020-09-0308:12ikitommia) lazy multi, with immutable values
[:multi {:dispatch :type, :children children-fn}]
b) mutable multi, backed by a custom (mutable) multimethod:
[:multi {:dispatch :type, :children my-multimethod}]
#2020-09-0308:12ikitommi… actually would be the same code, it’s in user-space whether to allow overriding the keys.#2020-09-0308:13ikitommishould not be many loc to implement#2020-09-0308:29jeroenvandijkThanks. Makes sense. I’ll try to adapt https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L796#2020-09-0308:40ikitommiIf you make a PR, would like that the default case (e.g. no :children key set) will not slow down -> the entry parsing will happen at schema creation time. for the case of dynamic childs - it would happen at runtime.#2020-09-0308:41ikitommione question is: what happens if you create a validator, explainer or generator out of that schema: should the current children be used or should those be dynamic too.#2020-09-0308:42ikitommie.g. if you add a branch after creating a validator, will the validators before that see it or not.#2020-09-0308:45jeroenvandijkWith clojure.spec I have one spec that contains all types. This gives you a suggestion in case the dispatch on type fails. E.g.
(s/def :cfn.all/Type #{"AWS::AmazonMQ::Broker" "AWS::AmazonMQ::Configuration" "AWS::ApiGateway::Account" "AWS::ApiGateway::ApiKey" "AWS::ApiGateway::Authorizer" "AWS::ApiGateway::BasePathMapping" "AWS::ApiGateway::ClientCertificate" .....})
This is not ideal either because it doesn’t have spell-check functionality. But to answer your question, I don’t think, at least for my use case, everything has to be dynamic
#2020-09-0314:39jeroenvandijk@ikitommi The start of this seems to be simple indeed https://gist.github.com/jeroenvandijk/59d22a726cda2158c01b9d63790aec50#file-malli_lazy-clj-L80 I’ve only added the validator part, not sure if the transformers and explainers will make things more painful#2020-09-0315:57ikitommi@jeroenvandijk just to Make sure: you do know all the possible dispatch keys in advance?#2020-09-0315:57ikitommi(if so, there might be a simpler solution)#2020-09-0316:17jeroenvandijkYeah all the dispatch types are known in this case. The raw schema data is close to 1mb. So that's the main reason to do it lazy#2020-09-0319:12ikitommi@jeroenvandijk This would be a small change in :ref impl:
(defn LazyRegistry [default-registry f]
  (let [cache* (atom {})
        registry* (atom nil)]
    (reset!
      registry*
      (mr/composite-registry
        default-registry
        (reify
          mr/Registry
          (-schema [_ name]
            (or (@cache* name)
                (do (println "loading" (pr-str name))
                    (when-let [schema (f name)]
                      (swap! cache* assoc name (m/schema schema {:registry @registry*}))
                      schema))))
          (-schemas [_] @cache*))))))

(def registry
  (LazyRegistry
    m/default-registry
    {"map1" [:map [:type [:= "map1"]] [:x :int]]
     "map2" [:map [:type [:= "map2"]] [:y :int]]
     "map3" [:map [:type [:= "map3"]] [:z :int]]}))

(m/validate
  [:multi {:dispatch :type}
   ["map1" [:ref "map1"]]
   ["map2" [:ref "map2"]]
   ["map3" [:ref "map3"]]]
  {:type "map3", :z 1}
  {:registry registry
   ::m/lazy-refs true})
;loading "map3"
;=> true
#2020-09-0319:14ikitomminew option :malli.core/lazy-refs that would control if the :refs are checked eagerly or lazily#2020-09-0319:15ikitommior there could be a :lazy variant of :ref to make things explicit.#2020-09-0319:16ikitommior a new property :lazy to :ref to mark it being lazy:
[:ref "map1"]

[:ref {:lazy true} "map1"]
#2020-09-0319:16ikitommiI think that’s actually good.#2020-09-0319:23ikitommihttps://github.com/metosin/malli/pull/252#2020-09-0320:05ikitommiactually, we can push all the changes from user api (e.f. schema props) into extender api (here: lazy registry impl). This allows to write fully lazy multis:
[:multi {:dispatch :type}
 "AWS::AmazonMQ::Broker"         
 "AWS::AmazonMQ::Configuration"
 "AWS::ApiGateway::Account"
 "AWS::ApiGateway::ApiKey"
 "AWS::ApiGateway::Authorizer"]
#2020-09-0320:07ikitommi(`:multi` uses the entry-syntax, like :map which allows single-value elements if they are valid schema reference types, now: just qualified keywords, should be strings too)#2020-09-0408:46ikitommi#2020-09-0408:47ikitommithat look ok @jeroenvandijk?#2020-09-0408:48jeroenvandijk@ikitommi it looks perfect! I need to wrap my head around it, but the demo looks exactly what I want. I’ll try to convert a spec project today and give some feedback#2020-09-0408:55ikitommigreat! changes in the core are in this commit: https://github.com/metosin/malli/pull/252/commits/3a5c7f35141e32eea3b374b6a81e706377a44d26, need to add polish the code before merging in, most likely this week.#2020-09-0416:38jeroenvandijkDoes Malli have an equivalent of (clojure.spec.alpha/map-of string? string) .e.g {"any-string-key" "any-string-value"} ?#2020-09-0416:51jeroenvandijkAs an alternative this seems to work:
(require '[malli.core :as m])

(defn map-of [k v]
  (clojure.walk/postwalk-replace
   {::k k
    ::v v}
   '[:fn (fn [x] (and (map? x)
                     (every? (fn [[k v]]
                               (::k k)
                               (::v v))
                             x)))]))

(m/validate (map-of 'string? 'string?) {:a 1})
#2020-09-0417:03ikitommi[:map-of string? string?]#2020-09-0417:04jeroenvandijkah 🙂#2020-09-0417:04jeroenvandijkThanks#2020-09-0417:15jeroenvandijkWould it make sense to add validation on the schema definition? e.g. i did this in a nested schema
(m/validate '[:map string? string?] {})
I had no idea of course and I was looking for the place that was causing this message
java.lang.UnsupportedOperationException: count not supported on this type: Symbol
#2020-09-0418:04ikitommiIntoSchema protocol will have a methods that describe both children and properties as malli schemas. After that, malli validates the malli schemas :) That requires the regex-schemas, which is the next thing. So, yes, but not yet.#2020-09-0418:06ikitomminot sure if there is an issue of that, should be.#2020-09-0417:39jeroenvandijkI was thinking of trying the non-lazy approach first, but I think the lazy version will be easier to debug 😅 I can print what faulty schema is being loaded just before it crashes#2020-09-0418:23jeroenvandijk@ikitommi So far the lazyness works really nice. I’ll let you know when I’m done#2020-09-0418:41jeroenvandijkIs it possible to somehow add the spellcheck on the dispatch as well? Would be super fancy 🙂 E.g. when I type
"AWS::AppSync::ApiK" ==> you misspelled "AWS::AppSync::ApiKey"
#2020-09-0418:43jeroenvandijkI’ve added a try/catch around the schema loading. Makes it much easier to debug a wrong definition:
(fn [type registry]
    (let [definition (lookup-fn type)
          schema (when definition
                   (try
                     (m/schema definition {:registry registry})
                     (catch Exception e

                       (throw (ex-info (str "Error while loading " type ": " (pr-str (ex-message e))) {:definition definition})))
                     ))]
      (println "loaded" (pr-str type))
      schema))
#2020-09-0419:22ikitommishould be possible. The spell checking reads the explain output. Just need to ensure :multi emits error on invalid dispatch key that the spell checker understands. Internally, :map and :multi both use the entry-syntax, so might be just few loc.#2020-09-0419:39jeroenvandijkAs far as I can see the spelling feature is focussed on the misspelling of keys and not of values. I probably missing something though. I was missing this spelling on values also when I was spec with expound and spell-check. Maybe it’s a different kind of problem#2020-09-0419:47jeroenvandijkah wait i think i see some pointers#2020-09-0419:24ikitomminext step would be to make clj-kondo use malli somehow, so we would get static inspection.#2020-09-0419:25ikitommiand the next would be to have auto-complete via lsp.#2020-09-0502:43rutledgepaulvI was wondering if lsp integration was in the cards 🙂 I would love to be able to write a malli schema and from it receive strong editor support for data that needs to conform to that schema (like some tools provide for xsd).#2020-09-0419:26ikitommi(no idea how to do the last, clj-kondo should be fun to try)#2020-09-0419:36jeroenvandijkSounds awesome!#2020-09-0420:01borkdude@ikitommi is this related to what you showed on ClojureD? I'm not sure what you mean by clj-kondo using malli#2020-09-0507:48ikitommiThe demo on ClojureD was schematized malli fn's emitting clj-kondo type definitions. I'll make a real impl out of that now with the clj-together funding. Not the first thing, but will most likely need help with that @borkdude, will poke you when about to do that. The second thing, which would be a nice experiment is if malli itself could be used inside clj-kondo to validate both malli data and schemas, e.g.
(do-it {:Type "AWS::AppSync::ApiKey"
        :Descriptionz "kikka"})
.. would emit clj-kondo errors:
{:ApiId ["missing required key"]
 :Descriptionz ["should be spelled :Description"]}
#2020-09-0507:51ikitommiautocomplete-stuff: I don't know how to do those, and most likely don't have to study that, but happy to help on Malli side if someone want's to try that out#2020-09-0508:27ikitommiwith the :multi dispatch key validation, it would yield:
(do-it {:Type "AWS::ApiGatway::UsagePlan"
        :Description "kikka"
        :UsagePlanName "kukka"}
.. would emit clj-kondo errors:
{:Type ["should be spelled \"AWS::ApiGateway::UsagePlan\""]}
#2020-09-0508:29borkdude@ikitommi I tried a similar thing using the type system in clj-kondo. It can be done, but it only works at places where literal maps are passed. So when you use assoc et al, it already breaks (although I think I have some logic which tries to account for that, it's been a while)#2020-09-0508:29borkdudeas an experiment, it'd be interesting what comes out of it#2020-09-0508:32borkdudeI recently listened to a podcast with Tony Kay. He is working on something similar called GuardRails Pro which will be closed source#2020-09-0508:33borkdudeIt would be great to have a free alternative#2020-09-0513:13ikitommiLooked up and listened the ClojureScript Podcast about GRP, sounds interesting.#2020-09-0513:14ikitommiInteresting times ahead, sounds like Clojure is getting mature :)#2020-09-0513:33ikitommi@jeroenvandijk small changes: • :dispatch is mandatory in :multi, e.g. can’t set via Schema creations opts => this makes it visible for error handling • malli.error/with-spell-checking now understands :multi dispatch values if the :dispatch value is a keyword#2020-09-0513:33ikitommihttps://github.com/metosin/malli/pull/252/commits/fecd792d02e20a9a22730ea0163b661c539eaa79#2020-09-0513:33ikitommi
(-> (m/explain
      Schema
      {:Type "AWS::AppSync::ApiKey"
       :Descriptionz "kikka"})
    (me/with-spell-checking)
    (me/humanize))
; loaded "AWS::AppSync::ApiKey"
; => {:ApiId ["missing required key"]
;     :Descriptionz ["should be spelled :Description"]}

(-> Schema
    (m/explain
      {:Type "AWS::ApiGatway::UsagePlan"
       :Description "kikka"
       :UsagePlanName "kukka"})
    (me/with-spell-checking)
    (me/humanize))
; => {:Type ["did you mean AWS::ApiGateway::UsagePlan"]}
#2020-09-0513:35ikitommifull gist here: https://gist.github.com/ikitommi/06143540e6259323b95253e87e1ef710#2020-09-0513:40jeroenvandijkWow, super nice. Thanks!#2020-09-0706:33ikitommifirst spike of heterogenous sequences, aka “regex schemas”. Looked into seqexp, clojure.spec and minimallist for inspiration. Slicing a [1 2 3 "4" "5" 6 7 8 9 "10"] sequence with spec-equivalient:
(s/cat :1 (s/+ int?)
       :2 (s/+ string?)
       :3 (s/+ int?)
       :4 (s/+ string?))
in malli, it would be:
[:catn
  [:1 [:+ int?]]
  [:2 [:+ string?]]
  [:3 [:+ int?]]
  [:4 [:+ string?]]]
or anonymus (uses vector indexes as keys)
[:cat
  [:+ int?]
  [:+ string?]
  [:+ int?]
  [:+ string?]]
some silly perf numbers with 10000 & time (and intermediate results):
Malli:
{:1 [1 2 3], :2 ["4" "5"], :3 [6 7 8 9], :4 ["10"]}
"Elapsed time: 35.535477 msecs"

Minimallist:
{:1 [1 2 3], :2 ["4" "5"], :3 [6 7 8 9], :4 ["10"]}
"Elapsed time: 386.464772 msecs"

Seqexp:
{:rest (), :match (1 2 3 "4" "5" 6 7 8 9 "10"), :1 (1 2 3), :2 ("4" "5"), :3 (6 7 8 9), :4 ("10")}
"Elapsed time: 686.08428 msecs"

Clojure.Spec:
{:1 [1 2 3], :2 ["4" "5"], :3 [6 7 8 9], :4 ["10"]}
"Elapsed time: 879.329458 msecs"
#2020-09-0905:38Vincent Cantin@ikitommi You made my day ! I never tried any benchmark on my lib, I did not know it was faster than Clojure Spec 😃#2020-09-1018:06ikitommiawesome#2020-09-0709:36borkdudecool :)#2020-09-0710:05borkdude@ikitommi I think this new clj-kondo feature can help with annotations generated by malli in some other config file: https://github.com/borkdude/clj-kondo/issues/992 Users can e.g. use {:config-paths ["malli-types"]}. If malli then spits out the type information in .clj-kondo/malli-types/config.edn, then it will work#2020-09-0710:05borkdudeThis is for the ClojureD-demoed functionality#2020-09-0710:06borkdudeI'm also considering a similar tool for clojure spec which inspects specs at runtime and then spits out a file. Maybe there could also be a CIDER middleware which populates things there as you eval code.#2020-09-0710:33borkdudebut if malli is going to have a defn like macro, I think making built-in functionality for it would also be cool. can we do this without depending on malli itself for keeping the dependencies of clj-kondo lean? e.g. it also doesn't depend on schema, it only recognizes the syntax#2020-09-0712:47ikitommi@borkdude good stuff. yes, the planned m/defn can emit the clj-kondo format directly, it’s a great start. Will keep an eye on your spec-tooling work too.#2020-09-0720:43tekacsReally looking forward to the m/defn support! I forked aave to add ClojureScript support and m/humanize-d errors recently and am adding in a few additional things — I’ll try to comment in the m/defn threads once they’re live to see whether any of the features added can make it across when the time comes. In particular, I’m binding up parameter and return types together (rather than separately), so that :multi / :or specs can be used as a poor-man’s (although runtime-dispatched) facsimile of overloading. i.e.
[[[:vector int?] => [:map-of int? string?]]
 [[:vector string?] => [:map-of string? int?]]]
translates to
[:or [:tuple [:vector int?] [:map-of int? string?]]
     [:tuple [:vector string?] [:map-of string? int?]]]
#2020-09-0720:43tekacscc/ @borkdude#2020-09-0720:45tekacsI’m also excited to use [:multi {:dispatch …}] instead of [:or] (the above are or-ed together), to see whether dispatch to a particular signature can be more sophisticated than simply ‘first matching signature’#2020-09-0806:11ikitommisounds great @tekacs, hopefully planning to push changes back to original Aave? Looking forward to hearing how the malli->kondo works with that. The. m/defn demo was a fork of Plumatic Schema s/defn, need to clean it up before use and open for comments. If the aave-syntax is already good, one option is just to add a fdef -malli decorator for existing functions into the core lib. Not sure. I believe Rich has been deep in the hammock thinking about new syntax for spec2 embedded defn syntax. Might take a while, but if becomes de-facto, malli could do that too.#2020-09-0806:14ikitommiPlumatic has btw s/conditional which might be handy with malli too. Like :multi but the keys are schemas to match the original value, e.g.
[:conditional
 [vector? [:vector int?]]
 [map? [:map [:x int?] [:y int?]]] 
#2020-09-0816:14tekacsoh that’s interesting, yes (to both :conditional and :else as really helpful things to have)#2020-09-0914:15jhacksFollowing along with the example here: https://github.com/metosin/malli#custom-registry
(def registry
  (merge
    (m/class-schemas)
    (m/comparator-schemas)
    (m/base-schemas)
    {:int (m/fn-schema :int int?)
     :bool (m/fn-schema :bool boolean?)}))

(m/validate [:or :int :bool] 'kikka {:registry registry})
; => false

(m/validate [:or :int :bool] 123 {:registry registry})
; => true
It looks like m/fn-schema no longer exists. I tried using m/-fn-schema (which does exist) but that didn’t work. How does this example work with the current code?
#2020-09-1015:28ikitommifixed the README#2020-09-0916:04ikitommi@jhacks oh, the README is out of sync. This should work:
(m/-simple-schema {:type :boolean, :pred boolean?})
#2020-09-0916:04ikitommithere seems to be (m/-predicate-schema :boolean boolean?) too#2020-09-0917:38jhacks@ikitommi Thanks, both methods work! Is there a way to register a default error message for schemas in the custom registry? For example, the custom :boolean schema has "unknown error" for the error message. Could I define a different message?#2020-09-1015:16ikitommi@jhacks currently no, but should be. wrote an issue about it: https://github.com/metosin/malli/issues/254 & will fix that soon.#2020-09-1015:19ikitommirenaming (& moving) the malli.error/ErrorSchema as malli.core/Humanized would make humanized errors more first class and would enable setting those easily when creating custom schemas - either via helpers of by reifying the protocols. Closure DCE will remove those anyway if not used in an app.#2020-09-1015:22ikitommicould also move the Generator protocol (not impls!) into core. Currently - if one want’s to implement fully custom Schma impl (like regal does), the lib must include all the support namespaces to access the Protocols. This drags is a lot of code that the end user might not need, making initial load time slower on clj and making the bundle size bigger on cljs.#2020-09-1017:50ikitommilooks legit.#2020-09-1018:00ikitommibig thanks to @vincent.cantin for the initial work on the issue! it’s now a full rewrite (protocols & vectors), but took the internal syntax from minimallist. minimallist inspired by malli, inspired by minimallist 😉#2020-09-1018:06ikitommiquite happy with the internal design. both validation and parsing use the same code, but when validating, the parsing results are not realized. that given example is 40µs on spec (both validation & parsing), malli is now: 6µs when parsing and 3µs on validation. not optimized yet, I think it could be made much faster still.#2020-09-1018:15ikitommi123 loc 🙂#2020-09-1023:48Vincent Cantinmicromallist#2020-09-1019:14borkdudelooks awesome#2020-09-1021:39tekacsI was considering making an issue to ask, but trying here first — is there a particular way you’d recommend doing a map on (or otherwise delegating to) existing schemas, @ikitommi? Some examples are: • Validating JS objects with a spec for their keys/values by using some of the implementation of [:map-of …] on a (decently efficient) cljs-bean.core/bean • ^ similarly with JS arrays • Ideally I’d like to pass the underlying explainer’s output through and be able to humanize the aggregate schema, which I’m especially having trouble with. So far I’ve done things like:
(defn transform-spec [transform inner-spec]
  (let [explainer (m/explainer inner-spec)]
    [:fn {:error/fn (fn [{:keys [value]} _] (explainer value))}
     #(m/validate inner-spec (transform %))]))

; and

(defn js-obj [& map-spec]
  (let [inner-spec (into [:map] map-spec)
        explainer (m/explainer inner-spec)]
    [:fn {:error/fn (fn [{:keys [value]}] (explainer value))}
     #(->> (bean %) (m/validate inner-spec))]))
… but I’m breaking the explainer output no matter how I slice it. The best approach for now I imagine is to directly implement the protocol(s) and do a bunch of delegation to the underlying schemas, but wondering if combinators/transformers are possible?
#2020-09-1022:52jhacks@ikitommi Thanks for the very informative response. https://github.com/metosin/malli/issues/254 looks great! Also, sequence/regex schemas! Wohoo!#2020-09-1023:44Vincent Cantin@ikitommi now, you can finally use Malli itself to validate Malli’s models.#2020-09-1109:50ikitommi@tekacs not 100% sure about your use case, what about using transformers?#2020-09-1523:38tekacsI was hoping rather to do something more like gen/fmap but for malli types. The hope being to be able to first run a transform on a value before validating it using an existing/builtin validator. I can do this today using simple-schema or similar, but passing errors along is just tricky.#2020-09-2015:12ikitommidoes the new -simple-schema solve this?#2020-09-1109:55ikitommiI recently tested the end-to-end performance of JSON and realized we could plug malli into the jsonista pipeline: define a malli schema, create a jsonista functional-decoder out of it which picks just the defined keys from the JSON stream and runs (already optimized) value transformation for it. As there are no intermediate conversions, it should be MUCH faster.#2020-09-1110:01ikitommi• old-style json: stream --json-decode--> edn --schema-decode--> domain-data --json-encode--> string --ring-adapter--> bytes • malli+jsonista: stream --json-and-schema-decode--> domain-data --json-and-schema-encode--> bytes#2020-09-1110:33borkdude@ikitommi That's exactly an issue that I posted a couple of years ago in the compojure-api channel :) Awesome that this will now be supported via malli!#2020-09-1110:34borkdudeThis is typically what you do in typed languages: write (de)serialization code which results in very efficient JSON processing. But in Clojure we typically just parse the entire blob#2020-09-1110:35borkdudeIf you have a working example of this, I'd be very interested#2020-09-1110:41borkdudesince jsonista is also based on jackson-core I might be able to switch/add from cheshire to jsonista without adding very much binary size. if we then also add malli and reitit and http-kit server ... in babashka I mean. small web-apps :)#2020-09-1114:55ikitommithat would be great! Would like to get vertx working with clj + graalvm at some point. After that, one could write a code that can run really fast with jvm, with low resources with graalvm and with bb for scripting.#2020-09-1114:57ikitommireitit+jsonista+porsas+pohjavirta is still one of the fastest jvm stacks in techempower db-query tests. It's a silly benchmark, but, still.#2020-09-1115:00ikitommihttps://www.techempower.com/benchmarks/#section=data-r19&amp;hw=ph&amp;test=db&amp;a=2&amp;f=35s1-1ekg-27xgcg-75-0-pfk-8vn0dc-kb4lg-13ydj4-8-0#2020-09-1115:01ikitommidon't have an example of the json+malli, just know it's possible. Will do.#2020-09-1523:38tekacsI was hoping rather to do something more like gen/fmap but for malli types. The hope being to be able to first run a transform on a value before validating it using an existing/builtin validator. I can do this today using simple-schema or similar, but passing errors along is just tricky.#2020-09-1315:49borkdudeI just realized that borkdude/dynaload isn't GraalVM friendly in that it uses require at runtime. I made an issue for this: https://github.com/borkdude/dynaload/issues/6 I think we'll have to adapt to the CLJS model of dynaload, i.e. the require is on the user, dynaload only prints a warning#2020-09-1316:13borkdudeNote: it did work, but it's just not good for binary size and compilation time#2020-09-1319:49borkdudeDo you test GraalVM builds on CI?#2020-09-1709:53StefanHi all, somebody pointed me to Malli (thx @borkdude @hobosarefriends) because of the situation with spec being alpha and spec2 on the horizon. But then I noticed in the Malli readme that it says: “Pre-alpha”, which scares me a bit. Can somebody clarify the situation and whether or not this is a good time to start using Malli? Our situation by the way is that we have a big existing codebase without any sort of spec/schema, so in that respect we’re starting from scratch.#2020-09-1711:11jeroenvandijkAll the solutions are officially still alpha. Trying any of them is a good learning experience. If you don’t have specific requirements, I would suggest to start with clojure.spec (1) as it has the most examples and documentation. I think Malli is modelled like spec(2) so any learnings with clojure.spec will be applicable later in malli#2020-09-1711:15borkdudeAlthough @ikitommi mentioned that the goal of the current Clojurists Together funding is to make a stable (non-alpha) release?#2020-09-1711:16borkdudeWith spec it remains to be seen. It's been in alpha for over 4 years.#2020-09-1711:28jeroenvandijkGood point 🙂#2020-09-1720:28rschmuklerThere's some discussion on github that might be interesting: https://github.com/metosin/malli/issues/207#2020-09-1711:02jeroenvandijk@ikitommi A small heads up I have generated a Malli spec that can validate all of our cfn templates (around 30, pretty big and complete). I’ll try to publish it soon and point to some pitfalls. One thing that is hard to debug are (deeply nested) recursive schemas. Without the use of [:ref …] it will give a stackoverflow error without a good pointer. And I had to do a trick in the :multi dispatch to prevent :invalid-type errors and to stay compatible with spell-check. All in all it works pretty well! Thanks#2020-09-1714:53jeroenvandijk@ikitommi Here is the current state of my AWS cloudformation malli code https://github.com/jeroenvandijk/aws.cloudformation.malli/blob/master/src/adgoji/aws/cloudformation/malli/validation.clj#L15-L232#2020-09-1714:54jeroenvandijk(now public)#2020-09-1714:56jeroenvandijkMaybe useful information; I’ve reduced the usage of :or and used :multi instead to prevent super long lists of errors (reduces branching factor)#2020-09-1714:57jeroenvandijkI think I broke proper spell check feedback by not using a keyword as dispatch function. I’ll look into this soon again#2020-09-1711:17ikitommi@stefan.van.den.oord Just checked that last bullet on “last things before initial stable release” yesterday. will clean up corners and will ship the alpha, most likely next week.#2020-09-1711:18ikitommithe public api has been mostly stable since june, there has been small changes in the advanced user / extender api, and most likely will be after the first release.#2020-09-1711:19borkdude@ikitommi (Note my remarks on borkdude/dynaload. I'm currently solving a problem with GraalVM. Using resolve at runtime bloats the binary with +20MB. But it does work.)#2020-09-1711:19ikitommi1.0.0 might be soon too, after feedback from the community.#2020-09-1711:20ikitommi@borkdude sorry, read you message, but didn’t have time to answer. There is no GraalVM tests atm, should be.#2020-09-1711:20borkdudeI will publish dynaload 0.2.0 or so that will have breaking changes to fix this problem, but I'll notify you#2020-09-1711:20borkdudeand possibly make a PR#2020-09-1711:21ikitommithat would be great! GraalVM support is top priority, don’t want bloated binaries 🙂#2020-09-1711:22ikitommiwhen writing libraries, the goals matter a lot. few years ago, didn’t know much about perf. can’t add that later. now, the cljs bundle size, learned how to handle that. Next: GraalVM as a target.#2020-09-1711:22borkdudeyeah. I have a dynaload-graal-friendly branch, with script/graal-test. currently this code still bloats:
#?(:clj (defn resolve* [sym]
          ;; TODO: this adds + 20MB to the GraalVM binary
          #_(let [ns (symbol (namespace sym))
                v (symbol (name sym))]
            (when-let [ns (find-ns ns)]
              (.getMapping ^clojure.lang.Namespace ns v)))))
so it's either in find-ns or in the namespace interop
#2020-09-1711:27borkdudeyeah, it's the namespace interop#2020-09-1711:28borkdudeI think using this interop will make GraalVM think it should hold on to more that it actually needs#2020-09-1711:30borkdudeso what we can do is only resolve at compile time. The user should take care of requiring the lib namespace before they load the dynaloaded code, else it will be considered not there.#2020-09-1711:32ikitommiis there a :preload thing with deps? other than the -e from command line?#2020-09-1711:33borkdudeno#2020-09-1711:33ikitommihave this on my project Justfile:
# start backend nrepl
@backend:
    clj -A:dev:test:common:backend -e "(require '[hashp.core])" -m nrepl.cmdline -i -C
#2020-09-1711:34borkdudefor graalvm builds with deps.edn that may work#2020-09-1711:35borkdudewith the lein + uberjar approach it may not#2020-09-1711:35borkdudejust take care in your main to require those libs first, then it will be solved. we can make this behavior graalvm only for example by reading an environment variable or java property#2020-09-1711:37ikitommiwhere is borkdude/am-i-in-graalvm library?#2020-09-1711:38ikitommiso: 1. cljs: preload or direct require 2. jvm: just in a classpath 3. graalvm: direct require , right?#2020-09-1711:38ikitommior: 2. jvm: just direct require#2020-09-1711:40ikitommican one differentiate if the lib is required or just in the classpath in 2? could be an option in dynaload?#2020-09-1711:41borkdude@ikitommi 1) I think in CLJS the order of require doesn't matter, because the check happens at runtime at the first deref. 2) this works because CLJ has runtime require 3) Like 1, but now the order matters, since we check at compile time, before the deref. About checking the classpath: not sure how this would look. That's kind of the same as compile time resolve.#2020-09-1711:44borkdudeChecking whether you are in a GraalVM binary already works, but that's too late. You have to check at (Clojure) compile time.#2020-09-1711:49borkdudeSo based on setting -J-Dborkdude.dynaload.target=graalvm-native we could alter the behavior or 2 to 3#2020-09-1711:50borkdudeor -J-Dborkdude.dynaload.resolve-time=compile#2020-09-1713:35eskosre: graalvm library, shouldn’t that be called antioch ?)#2020-09-1721:31borkdude@ikitommi Pushed dynaload 0.2.1 with better support for GraalVM binaries: https://github.com/borkdude/dynaload#graalvm#2020-09-1905:35ikitommiMalli now supports type-level properties (a breaking for extenders as there is a new protocol method in m/Schema) . Added a guide how to use them & also how to use non-registered Schemas.#2020-09-1905:35ikitommihttps://github.com/metosin/malli#custom-schema-types#2020-09-1905:39ikitommiIn short:
(def Over6
  (m/-simple-schema
    {:type :user/over6
     :pred #(and (int? %) (> % 6))
     :type-properties {:error/message "shuould be over 6"
                       :json-schema/type "integer"}}))

(m/into-schema? Over6)
; => true

(-> (m/explain Over6 5) (me/humanize))
; => ["shuould be over 6"]

(json-schema/transform [Over6 {:json-schema/example 7}])
; => {:type "integer", :example 7}
#2020-09-1912:45pithyless(m/explain Over6 "not-an-int") would not return the most obvious error. What is the idiomatic way to reuse and compose custom types? Something like this?
(def Over6
  [:and 
    pos-int? 
    (m/-simple-schema
     {:type :user/over6  ;; is this required?
      :pred #(> % 6)
      :type-properties {:error/message "should be over 6"}}])
#2020-09-1912:54pithylessI am currently using a custom version of -simple-schema (that allows setting custom error messages and generators). It's nice to see :type-properties added; maybe I can get rid of my custom code now. But I'm still struggling a bit with understanding how to best write custom types that don't need to re-implement predicate logic twice (once for validation, once for humanized errors); and how to compose things better.#2020-09-1912:56pithylessThis Over6 is a great example of a custom type, where ideally I could take advantage of all of the logic from e.g. pos-int? or [:int {:min 7}] and possibly add only additional :pred and :error/message#2020-09-1912:58pithylessOr I could just be wrong. But this is currently the one thing I don't understand on how to do well, in an otherwise fantastic library. :)#2020-09-1911:22schmeehello 👋 just curious if anyone has tried to derive Postgres schemas from Malli schemas (or Malli -> DB schemas in general)?#2020-09-1919:42dcjHere is what I did, This was my first Mailli use, not saying it is awesome or prefect: Added properties to map schema that provide info about how to defiine a Postgres table to store the map:
(def Datafile
  (m/schema
   [:map {:closed true
          :postgres/type :table
          :postgres/schema "slm"
          :postgres/table "datafile"
          :postgres/key-encoder (comp csk/->snake_case_keyword remove-trailing-? name)}
    [:id                {:optional true
                         :postgres/type :column
                         :postgres/datatype :bigserial
                         :postgres/key  :primary} int?]
    [:serial-number     {:postgres/type :column
                         :postgres/datatype [:varchar 64]
                         :postgres/null? false} string?]
    [:filename          {:postgres/type :column
                         :postgres/datatype  [:varchar 64]
                         :postgres/null? false} string?]
    [:extension         {:postgres/type :column
                         :postgres/datatype  [:varchar 8]
                         :postgres/null? false} string?]
    [:create-time       {:postgres/type :column
                         :postgres/datatype :timestamptz
                         :postgres/null? false} :zoned-date-time]
    [:ingest-time       {:optional true
                         :postgres/type :column
                         :postgres/datatype :timestamptz
                         :postgres/null? false} :zoned-date-time]
    [:complete?         {:optional true
                         :postgres/type :column
                         :postgres/datatype :boolean
                         :postgres/null? false} boolean?]
    ]
   {:registry registry}))
Then I wrote functions that create a DDL text file from this info. I also considered creating the table by connecting to the DB and directly creating it, but left that as an exercise for another day.... I also created functions that create a postgres-column-name-transformer And I created functions edn->postgres and postgres->edn that take a schema and a map, and returns a map that has been transformed per the schema. I found this pretty useful, and plan to continue to improve and use this going forward. There are cases where there is also some related web API, that has its own keys/values, and I did something similar for that, e.g. added properties that define the keys and datatypes of the API, and am able to create api->edn and edn->api functions to transform back and forth....
#2020-09-1919:53schmeecool, thanks for sharing! 😄#2020-09-2009:48ikitommi@pithyless good point about composition of things. Not sure myself what is a best practise with everything. Thinking aloud the ways to do things and when they should (or not) be used: Advanced usage: 1. implementing IntoSchema is the last effort in doing things, e.g. adding a new :regal/regex type, which has custom explain, transform etc. 2. using -simple-schema is helper for 1 (but only for leaf-schemas). Basic usage: using schemas as data should be the common way of doing & composing things. One can set transformation rules, humanized error messages, json schema mappings etc. as schema properties. Two options for composing things: 1. Vars. just say (def Over6 (m/schema [:int {:min 6, :description "should be over 6}])) and use the Var. Caveat: inlines the forms: (m/form [:and int? Over6]) ; => [:and int? [:int {:min 6, :description "should be over 6"}]], making large schema hard to read 2. Via registry:, e.g. {"Over6" [:int {:min 6}]} , keeps the reference visible: (m/form [:and int? "Over6"]) ; => [:and int? "Over6"] keeping the schema forms clean too#2020-09-2009:56ikitommiadded support for m/type-properties -based transformations too. And a way for -m/simple-schema to create the Schema Instances based on actual Schema instance properties.#2020-09-2009:59ikitommiNot sure how useful this is, but as the malli lifecycle already supports this, just made it easy to use:
(testing "with instance-based type-properties"
  (let [Over (m/-simple-schema
               (fn [{:keys [value]} _]
                 (assert (int? value))
                 {:type :user/over
                  :pred #(and (int? %) (> % value))
                  :type-properties {:error/message (str "should be over " value)
                                    :decode/string mt/-string->long
                                    :json-schema/type "integer"
                                    :json-schema/format "int64"
                                    :json-schema/minimum value}}))]

    (testing "over6"
      (let [schema [Over {:value 6}]]
        (testing "form"
          (is (= [:user/over {:value 6}] (m/form schema))))
        (testing "validation"
          (is (false? (m/validate schema 6)))
          (is (true? (m/validate schema 7))))
        (testing "properties"
          (is (= {:error/message "should be over 6"
                  :decode/string mt/-string->long
                  :json-schema/type "integer"
                  :json-schema/format "int64"
                  :json-schema/minimum 6}
                 (m/type-properties schema)))
          (is (= {:value 6}
                 (m/properties schema))))))

    (testing "over42"
      (let [schema [Over {:value 42}]]
        (testing "form"
          (is (= [:user/over {:value 42}] (m/form schema))))
        (testing "validation"
          (is (false? (m/validate schema 42)))
          (is (true? (m/validate schema 43))))
        (testing "properties"
          (is (= {:error/message "should be over 42"
                  :decode/string mt/-string->long
                  :json-schema/type "integer"
                  :json-schema/format "int64"
                  :json-schema/minimum 42}
                 (m/type-properties schema)))
          (is (= {:value 42}
                 (m/properties schema))))))))
#2020-09-2009:59ikitommi🍺 to the first who finds a valid use case for this.#2020-09-2010:01ikitommiso m/-simple-schema taks either a props map or a function of properties children => props so the props (including :type-properties) can be derived from schema properties.#2020-09-2016:51schmeeI’m trying out Malli for the first time:
(def schema [:map-of int? uuid?])
(def m {"0" "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
        "1" "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"})
(m/decode schema m mt/json-transformer)
user=> {"0" #uuid "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
        "1" #uuid "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}
I expected that this code would convert the keys to ints, can I modify the code in some way to make that happen?
#2020-09-2016:55ikitommihi @schmee. The mt/json-transformer doesn’t transform ints from strings as ints can be presented in JSON. But - actually, this only applies to values, so I think the keys should still be converted (as all keys are strings in JSON. Could you write an issue out of this?#2020-09-2016:56ikitommito make it work today, you can use mt/string-transformer which covers that out-of-the-box.
#2020-09-2016:56schmeeperfect, thanks! I’ll make an issue :thumbsup:#2020-09-2016:58schmeehttps://github.com/metosin/malli/issues/259#2020-09-2016:58ikitommire-wrote m/-predicate-schema, `m/-partial-predicate-schema` and `m/-leaf-schema` using m/-simple-schema. -39 loc.#2020-09-2016:59schmeefeel free to rename the issue to something that makes more sense to you!#2020-09-2018:05ikitommiActually, the fix was simple, we already had :map-of type transformation. Just needed add a optional argument to the mt/json-transformer that is the string-decoders, which are used for :map-of keys. The full impl looks like:#2020-09-2018:06ikitommi#2020-09-2018:08ikitommiand now:
(deftest map-of-json-keys-transform
  (let [schema [:map-of int? uuid?]]
    (doseq [data [{:0 "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
                   :1 "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}
                  {"0" "2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
                   "1" "820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}]]

      (is (= {0 #uuid"2ac307dc-4ec8-4046-9b7e-57716b7ecfd2"
              1 #uuid"820e5003-6fff-480b-9e2b-ec3cdc5d2f78"}
             (m/decode schema data mt/json-transformer))))))
#2020-09-2018:09ikitommi@schmee fixed in master.#2020-09-2019:01schmeehaha, that is one fast fix, thanks! 😄#2020-09-2019:01schmeelooking forward to playing around more with Malli :thumbsup:#2020-09-2208:13steveb8nI just started using Malli for json-schema. this lib is becoming a swiss army knife for me 🙂#2020-09-2208:54ts1503Hello guys. Any ideas when a stable version of malli will arrive? I’m using the malli for a while in production but the latest release broke my app because the transitive dependency doesn’t work with Java 11 (https://github.com/borkdude/sci) 😢#2020-09-2208:56borkdude@sergey.tkachenko sci is optional with malli since a couple of releases, so if you aren't using it, it should not be a problem#2020-09-2208:57borkdudeif you are using it, then there are instructions in the README of sci what should be done with java 11#2020-09-2208:57ts1503I’m not using it directly I’m using only a malli#2020-09-2208:58borkdudethen depending on a newer version of malli should automatically fix this#2020-09-2208:58ts1503what is the latest version? I see only a 0.0.1-SNAPSHOT#2020-09-2208:59ikitommi0.0.1-20200920.124124-25#2020-09-2208:59ts1503and it worked fine for a months but after the latest release/override it broke my build#2020-09-2208:59ikitommiall SNAPSHOTs have immutable versions behind them.#2020-09-2209:00ikitommithat version has these deps:
<dependencies>
    <dependency>
      <groupId>borkdude</groupId>
      <artifactId>dynaload</artifactId>
      <version>0.1.0</version>
    </dependency>
    <dependency>
      <groupId>borkdude</groupId>
      <artifactId>edamame</artifactId>
      <version>0.0.11-alpha.13</version>
    </dependency>
    <dependency>
      <groupId>org.clojure</groupId>
      <artifactId>test.check</artifactId>
      <version>1.0.0</version>
    </dependency>
  </dependencies>
#2020-09-2209:00ts1503is this version (0.0.1-20200920.124124-25) in the clojars?#2020-09-2209:01ikitommiso, you can use the “previous” SNAPSHOT too, just need to find the full version, e.g. .m2/repository/metosin/malli/0.0.1-SNAPSHOT#2020-09-2209:01ikitommiyes#2020-09-2209:01ikitommi
➜  0.0.1-SNAPSHOT ls -l
total 304
-rw-r--r--  1 tommi  staff    405 Sep 21 08:02 _remote.repositories
-rw-r--r--  1 tommi  staff  19391 Sep 21 08:02 malli-0.0.1-20200305.102752-13.jar
-rw-r--r--  1 tommi  staff     40 Sep 21 08:02 malli-0.0.1-20200305.102752-13.jar.sha1
-rw-r--r--  1 tommi  staff   1692 Sep 21 08:02 malli-0.0.1-20200305.102752-13.pom
-rw-r--r--  1 tommi  staff     40 Sep 21 08:02 malli-0.0.1-20200305.102752-13.pom.sha1
-rw-r--r--  1 tommi  staff  24437 Sep 21 07:37 malli-0.0.1-20200715.082439-21.jar
-rw-r--r--  1 tommi  staff     40 Sep 21 07:37 malli-0.0.1-20200715.082439-21.jar.sha1
-rw-r--r--  1 tommi  staff   1692 Sep 21 07:37 malli-0.0.1-20200715.082439-21.pom
-rw-r--r--  1 tommi  staff     40 Sep 21 07:37 malli-0.0.1-20200715.082439-21.pom.sha1
-rw-r--r--  1 tommi  staff  27727 Sep 20 15:43 malli-0.0.1-20200920.124124-25.jar
-rw-r--r--  1 tommi  staff     40 Sep 20 15:43 malli-0.0.1-20200920.124124-25.jar.sha1
-rw-r--r--  1 tommi  staff   1547 Sep 20 15:43 malli-0.0.1-20200920.124124-25.pom
-rw-r--r--  1 tommi  staff     40 Sep 20 15:43 malli-0.0.1-20200920.124124-25.pom.sha1
-rw-r--r--  1 tommi  staff  24437 Sep 21 07:37 malli-0.0.1-SNAPSHOT.jar
-rw-r--r--  1 tommi  staff   1692 Sep 21 07:37 malli-0.0.1-SNAPSHOT.pom
-rw-r--r--  1 tommi  staff    765 Sep 20 15:43 maven-metadata-clojars.xml
-rw-r--r--  1 tommi  staff     40 Sep 20 15:43 maven-metadata-clojars.xml.sha1
-rw-r--r--  1 tommi  staff    370 Sep 20 15:43 resolver-status.properties
#2020-09-2209:01ikitommi… should be.#2020-09-2209:01borkdudeI used to have a way of seeing these fully qualified snapshot versions on clojars. forgot#2020-09-2209:01borkdudeI usually look it up in CI when it is pushed#2020-09-2209:01ts1503there is only a snapshot#2020-09-2209:02ts1503or I’m missing something?#2020-09-2209:03borkdudebtw @ikitommi please bump dynaload to 0.2.2 :)#2020-09-2209:04ikitommiwill do + publish a new version out.#2020-09-2210:02ikitommiupdated dynaload and pushed to clojars, ping @sergey.tkachenko#2020-09-2210:02ikitommi
Downloading: metosin/malli/0.0.1-SNAPSHOT/malli-0.0.1-20200922.100025-26.pom from clojars
#2020-09-2210:02ikitommithat’s the immutable version handle to it.#2020-09-2210:11ts1503Thanks#2020-09-2302:24Vincent Cantin@ikitommi This issue might be interesting for Malli, specially before the release. https://github.com/green-coder/minimallist/issues/8#2020-09-2305:16ikitommi@vincent.cantin that is the case of mixing :map and :map-of right? the issue and discussion in malli-side here: https://github.com/metosin/malli/issues/43#2020-09-2306:51ikitommidiscussion about dates, need this (date range validation) in a work project, comments welcome: https://github.com/metosin/malli/issues/49#2020-09-2306:57ikitommia good backgrounder: http://widdindustries.com/ecma-temporal-vs-java-time/#2020-09-2323:43dcjI have some code I wrote a few months back that made use of map-entries which is now gone... IIRC, map-entries returned a collection of [key properties value-schema] ? AFAICT in my old code, I would get this, and access the key or the properties, what should I do to move my code to entries? What exactly is a -val-schema?#2020-09-2403:48ikitommi@dcj there is poor man's documentation about that change in CHANGELOG. The m/children returns the [key properties schema] tuple3s, so just use that instead. The new m/entries (renamed iit so it's not a silent/evil breakage) is needed to effectively use the entry properties in schema applications like value transformation and schema transformations.#2020-09-2404:02ikitommithere is a long discussion about the options and the second in the related issue (https://github.com/metosin/malli/pull/212) and it's also documented in docstrings now: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1127-L1155#2020-09-2405:48Dave SimmonsMorning - I've been using Malli for the last few month. Last night when compiling my code lein pulled down a new version. I now get a failure when trying to compile my project Exception in thread "main" Syntax error compiling at (malli/core.cljc:1164:1).#2020-09-2405:49Dave SimmonsI pulled down an earlier version of my project but in case I'd done something daft but still get the same problem. Has anyone else experienced this issue? cheers.#2020-09-2406:11ikitommi@shortlyportly i can reproduce. the new dynaload works differently (better!), will fix that.#2020-09-2406:21Dave Simmons@ikitommi - awesome - many thanks.#2020-09-2406:32ikitommi@shortlyportly should be fixed in master & [metosin/malli "0.0.1-20200924.063109-27"], ping @sergey.tkachenko too.#2020-09-2406:36ikitommihttps://github.com/metosin/malli/pull/261#2020-09-2406:35ikitommialso, sci is now loaded when it’s first used. Using preload (cljs) or direct require (cljs, graalvm, jvm) makes it eager. But: having sci on classpath and not needing it, means it wont get loaded by malli, yielding faster startup time:
(time (require '[malli.core :as m]))
"Elapsed time: 466.76624 msecs"
=> nil
(time (m/eval "(+ 1 1)"))
"Elapsed time: 1591.515317 msecs"
=> 2
(time (m/eval "(+ 1 1)"))
"Elapsed time: 0.587728 msecs"
=> 2
(time (m/eval "(+ 1 1)"))
"Elapsed time: 0.772357 msecs"
=> 2
#2020-09-2406:35ikitommiwhen clj command line tooling 3rd party library aot caching works, this can be made eager again.#2020-09-2407:01Dave Simmons@ikitommi - thank you - seems to be back and working. cheers#2020-09-2416:29dcjNext change, m/fn-schema went away? How should I change the below....
(def registry
  (merge (m/predicate-schemas)
         (m/class-schemas)
         (m/comparator-schemas)
         (m/base-schemas)
         {:zoned-date-time (m/fn-schema :zoned-date-time #'zoned-date-time?)
          :local-date      (m/fn-schema :local-date      #'local-date?)}))
#2020-09-2416:39ikitommi@dcj there were 4 variants of the same thing, fn-schema, leaf-schema, predicate-schema, partial-predicate-schena , all replaced by m/-simple-schema. Also, there is way to reset the default registry now. But the schema:
(m/-simple-schema {:type :local-date, :pred #'local-date?})
#2020-09-2416:49ikitommiit also give you a way to read the schema properties and use them for the validation or transformation:
(def Date
  (m/-simple-schema 
    (fn [{:keys [format type min]} children]
      ;; check props and children here, at schema instance creation time
      (let [string->date (create-formatter-somehow format)
            min-date (string->date min]
        {:type :local-date
         :pred (fn [x] (and (instance? type x) (check-that-is-greater-than min))
         :type-properties {:decode/string string->date
                           :error/message {:en "should be date"}
                           :gen/gen generator-for-date}})))

(def LocalDateThisYear
  [Date {:format "YYYY-MM-DD", :min "2020-01-01", :type java.time.LocalDate}])

(m/validate LocalDateThisYear #time/local-date "2020-12-12") ; => true

(m/decode LocalDateThisYear "2020-12-12" mt/string-decoder) ; => #time/local-date "2020-12-12"
#2020-09-2416:52ikitommi
(def registry
  (merge (m/-predicate-schemas)
         (m/-class-schemas)
         (m/-comparator-schemas)
         (m/-base-schemas)
         (m/-type-schema) ;; <--- new too
         {:zoned-date-time (m/-simple-schema {:type :zoned-date-time, :pred #'zoned-date-time?})
          :local-date      (m/-simple-schema {:type :local-date,      :pred #'local-date?})}))
#2020-09-2416:54dcj@ikitommi: Thank you for all this help/context/insight!#2020-09-2419:01ikitommiadded tests and merged the lazy registries & lazy multi. Here’s the final api:
(def registry
  (mr/lazy-registry
    (m/default-schemas)
    (fn [type registry]
      ;; simulates pulling CloudFormation Schemas when needed
      (let [lookup {"AWS::ApiGateway::UsagePlan" [:map {:closed true}
                                                  [:Type [:= "AWS::ApiGateway::UsagePlan"]]
                                                  [:Description {:optional true} string?]
                                                  [:UsagePlanName {:optional true} string?]]
                    "AWS::AppSync::ApiKey" [:map {:closed true}
                                            [:Type [:= "AWS::AppSync::ApiKey"]]
                                            [:ApiId string?]
                                            [:Description {:optional true} string?]]}]
        (println "... loaded" type)
        (some-> type lookup (m/schema {:registry registry}))))))

;; lazy multi, doesn't realize the schemas
(def CloudFormation
  (m/schema
    [:multi {:dispatch :Type, :lazy-refs true}
     "AWS::ApiGateway::UsagePlan"
     "AWS::AppSync::ApiKey"]
    {:registry registry}))

(m/validate
  CloudFormation
  {:Type "AWS::ApiGateway::UsagePlan"
   :Description "laiskanlinna"})
; ... loaded AWS::ApiGateway::UsagePlan
; => true

(m/validate
  CloudFormation
  {:Type "AWS::ApiGateway::UsagePlan"
   :Description "laiskanlinna"})
; => true
#2020-09-2515:19jeroenvandijkThank you!#2020-09-2419:03ikitommihas also the :multi key spell-checker:
(deftest multi-error-test
  (let [schema [:multi {:dispatch :type}
                ["plus" [:map [:value int?]]]
                ["minus" [:map [:value int?]]]]]

    (is (= {:type ["invalid dispatch value"]}
           (-> schema
               (m/explain {:type "minuz"})
               (me/humanize))))

    (is (= {:type ["did you mean minus"]}
           (-> schema
               (m/explain {:type "minuz"})
               (me/with-spell-checking)
               (me/humanize))))))
#2020-09-2515:15borkdudeNow I remember how to see all the immutable SNAPSHOTS: http://repo.clojars.org/metosin/malli/0.0.1-SNAPSHOT/ @ikitommi#2020-09-2515:33dcjHow would one specify one of these immutable snapshots in a lein project.clj? I attempted to do exactly that earlier this week, but failed miserably....#2020-09-2515:36borkdudee.g. [metosin/malli "0.0.1-20200924.063109-27"]#2020-09-2517:06kwrooijenVery handy. Thanks!#2020-09-2515:23ikitommigreat, thanks!#2020-09-2910:05ikitommiI must have asked this before, but is there an example how to add tests “works on GraalVM too”?#2020-09-2910:16ikitommiCurrent:
(m/schema [:tuple int? int?])
; => [:tuple int? int?]
Should it be:
(m/schema [:tuple int? int?])
; => #malli/schema[:tuple int? int?]

(m/form (m/schema [:tuple int? int?]))
; => [:tuple int? int?]
GOOD: • schemas look like schemas, not forms BAD: • can’t pr-str a schema and read it back with m/schema, would need malli.edn/read-string
#2020-10-0223:07dcjIs there a path from Swagger/JSON-Schema -> Malli? I know there are paths the other way.... I don't expect this would be perfect and require no thought, but imagine I encounter something described by Swagger/JSON-Schema I need/want to deal with from Clojure, Some sort of malli translator that generated a best-effort malli schema, and also included the "and here is how to get back to JSON-Schema/Swagger" properties, and then I would edit away on the Clojure/Mailli side of the schema to make it how I want...#2020-10-0409:14ikitommi@dcj not yet, but there is https://github.com/metosin/malli/pull/211 and I recall other people trying that too. Don't have time to dig in to this yet, but would love to get that to work with malli#2020-10-0409:16ikitommisomeone was doing this for spec and I believe https://github.com/oliyh/martian has partial impl for plumatic schema.#2020-10-0700:51steveb8nQ: is there a typo in the readme in the custom registry section? :post-int#2020-10-0700:58steveb8nAnother one: is m/s-simple-schema a typo? This fn doesn’t exit#2020-10-0704:45ikitommiwelcome @kari.marttila! first one definitely a typo, could not find the latter. But PR welcome to fix any typos.#2020-10-0704:53steveb8nok. I’ll make a PR for both#2020-10-0704:55steveb8nscratch that. I don’t know what you would use to replace the second one. I’ll PR for the typo#2020-10-0704:55steveb8nhere’s the second one…
Custom registry
Example to create a custom registry without the default core predicates and with :bool and :pos-int Schemas:

(def registry
  (merge
    (m/class-schemas)
    (m/comparator-schemas)
    (m/base-schemas)
    {:bool (m/-simple-schema {:type :bool, :pred boolean?})
     :pos-int (m/-simple-schema {:type :pos-int, :pred pos-int?})}))

(m/validate [:or :bool :pos-int] 'kikka {:registry registry})
; => false

(m/validate [:or :bool :post-int] 123 {:registry registry})
; => true
#2020-10-0704:56steveb8nm/-simple-schema doesn’t exist as a fn. I’m not sure what you would use instead#2020-10-0705:10ikitommihttps://github.com/metosin/malli/blob/master/src/malli/core.cljc#L211#2020-10-0705:10ikitommiare you using the latest version?#2020-10-0705:47steveb8nah, no. I’m not. sorry. my mistake#2020-10-0712:52ikitommiok, plan is to ship 0.1.0 tomorrow. going to clean up some internals and polish some docs, but anything relevant missing? the GraalVM tests? https://github.com/metosin/malli/pull/272#2020-10-0712:53ikitommialso, removing :list as it’s not very useful, right?#2020-10-0807:52steveb8nI haven’t used it fwiw#2020-10-0807:09ikitommi#2020-10-0808:47borkdudeWhat's the binary size?#2020-10-0809:57ikitommi18mb, with sci#2020-10-0810:00borkdudeSounds about right #2020-10-0810:00ikitommi11mb without#2020-10-0810:00borkdudeAlso sounds right!#2020-10-0817:17ikitommi[metosin/malli "0.1.0"] • clojars • clojureverse • metosin blog • #malli • reddit • twitter • #announcements #2020-10-0817:26ikitommiok, thank you everyone, Malli is officially out! if there are more places to share, please do.#2020-10-0818:03dcjCongrats! And thank you for your amazing work on this!#2020-10-0906:49ikitommithanks!#2020-10-0817:58dharriganIs it still alpha?#2020-10-0818:08borkdude@dharrigan I think it went from pre-alpha to alpha now. So in Clojure world, it's production time? 😆#2020-10-0818:09dharriganYeah, that's what I think. I think, given what happened a few weeks/months ago, about the push from the core team to move away from eternal alpha to better versions, and given that malli is now out, then perhaps alpha isn't the right term here?#2020-10-0818:09dharriganIt's not even beta anymore#2020-10-0818:11borkdudeWhat move away from eternal alpha? spec has been in alpha since it's existence#2020-10-0818:13dharriganYeah, there are exceptions of course#2020-10-0818:14dharrigan#2020-10-0818:14dharriganthere you go#2020-10-0818:15dharriganAfter that blog post on Febrary, quite a few libraries moved away from alpha, the eternal alpha.#2020-10-0818:16dharriganFrom first-hand experience trying to introduce Clojure (and libraries into the application), seeing things as alpha can be off-putting to others in the teams.#2020-10-0818:16dharrigan.#2020-10-0820:45katoxIt's Clojure we use it in production since June. And it's great.#2020-10-0821:06dharriganI've been using malli now for a few months, stable, featureful and works brilliantly 🙂#2020-10-0821:17schmee@ikitommi is it of interest to have a Schema -> Postgres DDL transformation in Malli? 🙂#2020-10-0821:21dcjI'm interested.... I have a simple prototype that handles only what I need, and "works for me"....#2020-10-0822:01kwrooijenI actually wrote a library that does that https://github.com/kwrooijen/gungnir#2020-10-0822:01kwrooijenIn combination with HoneySQL / next.jdbc#2020-10-0822:02kwrooijenIt allows you to define your table structure, with transformations / validations, using Malli#2020-10-0822:09dcj@UG9U7TPDZ Wow, that looks cool! I will definitely try that out.#2020-10-0823:19macIs it correctly understood that malli does not support sets as predicates? if so is there then an alternative?#2020-10-0900:06schmee@mac the equivalent is [:enum ...]#2020-10-0906:38mac@schmee Ah, thanks.#2020-10-0906:55ikitommi@dharrigan could have been a non-alpha, but was busy getting it out, so just alpha now. Grand goal is to get 1.0.0 out as soon as possible, after the final pieces (sequences, parsing, functions, default options) have been implemented, might effect the public api so that need to bump MAJOR. Most likely something that is easily migrated from pre-1.0.0.#2020-10-0907:00ikitommi@mac there is a special type-schemas mechanism which would allow using using plain sets, but having too much shortcuts might make things harder to understand. There is already a such type shortcut for regexs, not sure if that was the right call:
#"\d+kikka"        ;; using the type-shortcut
[:re #"\d+kikka"]  ;; explicit
sets - besides :enum, there is :fn for any function.
[:enum "a" "b" "c"]
[:fn #{"a" "b" "c"}]
:enum doesn’t do type inferring atm, so you can hint the type so that JSON Schema, value transformation etc. work correctly:
[:and keyword? [:enum :kikka :kukka]]
#2020-10-0907:02dharrigan@ikitommi fantastic! Thank you for the update (and also the libraries, reitit and malli - super awesome sauce!)#2020-10-0907:23mac@ikitommi Thanks, the reg-ex shortcut looks pretty idiomatic to me.#2020-10-0909:09borkdudeAre there any libs like cli-matic that use malli instead of spec for arg parsing/validation?#2020-10-0912:18Toni VanhalaToday was supposed to be ClojuTRE 2020, but what we got was t-shirts with malli validation errors. Just put these fun little items to Metosin shop: https://shop.spreadshirt.net/metosin/#2020-10-0917:57Lucas Félixhi! - sorry for my english - is there something like spec functions (https://clojure.org/guides/spec#_specing_functions) in malli? if not, what’s the motivation?#2020-10-0918:22borkdude@lfelixsampaio https://github.com/metosin/malli/issues/125#2020-10-0918:24Lucas Félixthanks! @U04V15CAJ#2020-10-0922:56steveb8n@ikitommi I’m curious why you’ve stopped mentioning the multi-tenant validation capabilities of Malli. for me, this was the original reason for using it and the registry design is very well done#2020-10-1007:48borkdude@ikitommi I'm reading through the Malli README and found a couple of spelling improvements: Homogenous -> Homogeneous#2020-10-1007:49borkdudeAnd this sentence: > You can also decomplected maps keys and values using registry references. seems to be grammatically a bit off#2020-10-1008:13ikitommi@borkdude fixes are most welcome.#2020-10-1008:13borkdudeJust passing it here, do what you want with it :)#2020-10-1008:15ikitommiwill fix#2020-10-1008:17ikitommifixed#2020-10-1008:18ikitommitoyed with the declarative schema transformations, which might be useful when defining schemas in EDN:
(require '[malli.core :as m])
(require '[malli.util :as mu])
(require '[malli.error :as me])

(def registry (merge (m/default-schemas) (mu/schemas)))

(def XZ
  (m/schema
    [:select-keys
     [:merge
      [:map [:x int?]]
      [:map [:y int?]]
      [:map [:z int?]]]
     [:x :z]]
    {:registry registry}))

XZ
; [:select-keys
;  [:merge
;   [:map [:x int?]]
;   [:map [:y int?]]
;   [:map [:z int?]]]
;  [:x :z]]

;; get the effective schema
(m/deref XZ)
; [:map [:x int?] [:z int?]]

;; internally uses the pre-computed effective schema
(-> XZ
    (m/explain {:x 1})
    (me/humanize))
; {:z ["missing required key"]}
#2020-10-1008:21borkdudeI was wondering, does malli also do the destructuring that spec does e.g. on a sequential regex schema? (s/cat etc)#2020-10-1008:21borkdudeor s/or, etc#2020-10-1008:22borkdudesomeone should probably write a comparison/migration page for malli <-> spec :)#2020-10-1008:22ikitomminot yet, the internal api works, but not integrated into malli.core: https://github.com/metosin/malli/issues/180#2020-10-1008:23ikitommi… and https://github.com/metosin/malli/issues/241 after that.#2020-10-1008:24borkdudeexactly! thanks!#2020-10-1018:53ziltiWhen creating a validator, how do I use non-core functions in a :fn? Specifically, I want to use clojure.string/blank?#2020-10-1019:48ikitommi@zilti :fn takes any function as unquoted, so [:fn clojure.string/blank?]. They don't serialize correctly, but work on the same runtime. If you use sci, you need to add custom bindings for the function for all runtimes. I believe str/blank? is part of sci default bindings, so [:fn 'str/blank?] should work too.#2020-10-1019:49ikitommialso, you can say [:string {:min 1}].#2020-10-1019:49borkdudeclojure.string/blank? works in sci by default#2020-10-1019:49ziltiOh, that :min seems to be the most straightforward for my usecase, I'll use that!#2020-10-1019:52ikitommi:string has good default error messages: https://github.com/metosin/malli/blob/master/test/malli/error_test.cljc#L262-L283#2020-10-1110:21Lucy WangFixed a bunch of typos https://github.com/metosin/malli/pull/275#2020-10-1111:00ikitommimerged, thanks!#2020-10-1112:10ikitommiwould like to make sci explicitly optional. either via a flag (non-breaking) or via explicit option (breaking, but for the better): https://github.com/metosin/malli/issues/276. Two days ago would have just done the latter, but now malli is released and goal has been not to break things. What do you think? save the breaking change for 1.0.0? just do it? something else?#2020-10-1112:13borkdude@ikitommi Maybe sci could be one of many possible evaluators?#2020-10-1202:33steveb8nI think a breaking change for the 1.0 release is ok. We know you plan to accrete only from 1.0#2020-10-1202:34steveb8noptional sci is great for the browser use case. making that the default is a good idea imho#2020-10-1202:34steveb8nI’m gonna use it server side also#2020-10-1205:48ikitommigood reason not yet to read sci-powered schemas from untrusted sources: https://github.com/borkdude/sci/issues/348#2020-10-1205:48ikitommithat's now linked in Malli Readme.#2020-10-1112:13borkdudelike clojure.core/eval is another one#2020-10-1112:13borkdudeand maybe the user should specify that#2020-10-1112:14borkdudebreaking would be ok I think#2020-10-1112:14borkdudesince the project is explicitly alpha#2020-10-1112:14ikitommiyes, the option1 would allow that:
(require '[malli.sci :as ms])
(require '[malli.core :as m])

(def options
  {:evaluator (ms/evaluator)
   :registry (ms/default-registry)})

(def Schema (m/schema [:fn '(fn [x] (string? x))] options))

(m/validate Schema "kikka")
; => true
#2020-10-1112:15ikitommi.. would also remove need of :preloads etc, as you actually need to require the code to make it work.#2020-10-1112:15borkdudeI think that's reasonable as long as it's documented well#2020-10-1112:16borkdudeand keep a list of breaking changes in CHANGELOG.md#2020-10-1112:18borkdudemalli.sci would still require sci for you right. so then there's no need for preloads#2020-10-1112:19ikitommialso, currently you can swap the default registry using the JVM/clj-compiler options, but not the default options. if you want to enable sci globally, that should be changed to swap the default options. I did a spike on that, but had too much open issues to think that through. Now, it seems like it would have been a right call.#2020-10-1112:19ikitommihttps://github.com/metosin/malli/pull/235#2020-10-1112:21ikitommithat would be big breaking change, but something that could be documented and migrated easily too.#2020-10-1112:21ikitommimaybe all roads lead for quick 1.0.0 🙂#2020-10-1112:22ikitommithanks for you thoughts on this.#2020-10-1114:00motformHi! does anyone have any experience using malli to validate/define the re-frame app-db?#2020-10-1122:13steveb8n@love.lagerkvist I’m doing this for forms currently https://github.com/stevebuik/fork-malli-ideas#2020-10-1122:14steveb8nworks great. have not moved the app-db level validations from spec to Malli yet but I certainly plan to do so#2020-10-1203:55Lucy Wang@love.lagerkvist I'm using re-frame.core/reg-global-interceptor to verify the app-db matches my schema in every change, and prints an console warning if it doesn't conform. But re-frame.app-db is just a normal map (live inside an reagent.atom), so it's nothing special when using it with malli IMO.#2020-10-1205:35ikitommimerged #277, non-breaking disabling & configuration of sci. Will make a 1.0.0 issue with suggestion to make swappable & explicit evaluator. Thanks @borkdude @steveb8n#2020-10-1207:00borkdude@ikitommi note that termination safe does have impact on performance of realizing seqs (since it’s checked). I’m not sure if the issue about execution time is fundamentally solvable.#2020-10-1208:44ikitommiIt is one of the harder problems in comp.sci ;) With Malli & sci, I concider safety as more important. Following the issue & pr on sci-side, happy to incorporate anything that is found for this.#2020-10-1208:51borkdudeI think the fundamental solution would be to run the sci expression in a thread on the JVM and kill it if it takes too long or in a webworker in the browser and do the same.#2020-10-1208:51borkdudeMaybe I should drop the termination-safe option as well - not sure.#2020-10-1207:54motform@wxitb2017 that sounds about right, I’ll do that. However, I was also thinking if it was possible to co-locate/generate/properly infer an inital app-db directly from malli. It would be really nice to have a single source of truth, kind of like a reitit router with views, coercion and controllers. (it’s probably super possible and I have to dig more into malli or a very bad idea)#2020-10-1215:15ikitommiYou could have a Malli schema defined for the app-db, with default and create the initial state from those, e.g.
(defn create [?schema]
  {:validator (m/validator ?schema)
   :explainer (m/explainer ?schema)
   :initial (m/decode ?schema nil (mt/default-value-transformer))})

(create [:map {:default {}}
         [:user [:map {:default {}}
                 [:first-name [:string {:min 1, :default ""}]]
                 [:last-name [:string {:min 1, :default ""}]]]]])
;{:validator #object[...],
; :explainer #object[...],
; :initial {:user {:first-name "", :last-name ""}}}
inferring is powerful, but not fully accurate, e.g. is something a vector of things of a tuple of always 2.
#2020-10-1215:16ikitommi(the mt/default-transformer could have an option to create empty maps by default…)#2020-10-1307:26ikitommihttps://github.com/metosin/malli/pull/278, added :key and :defaults to mt/default-value-transformer to make it easier to create empty/default values with it:
(m/decode
  [:map
   [:user [:map
           [:name :string]
           [:description {:ui/default "-"} :string]]]]
  nil
  (mt/default-value-transformer
    {:key :ui/default
     :defaults {:map (constantly {})
                :string (constantly "")}}))
; => {:user {:name "", :description "-"}}
#2020-10-1307:27ikitommithe values in :defaults can access the Schema instance, so can use any information from it to create the default, e.g. schema -> default function.#2020-10-1308:36dangercoderIs it possible to override the default error message for :re?#2020-10-1309:06kwrooijen
[:re {:error/message "Invalid email"}
     #"
Something like this?
#2020-10-1309:32dangercoderI would like it to be general#2020-10-1309:32dangercoderLike, for all regexp errors I apply a custom message#2020-10-1309:36kwrooijenAh like that. I'm not familiar with a solution for that#2020-10-1310:58ikitomminot a good way to do that, but one can override the :errors from malli.error like this:
(-> #"\d+"
    (m/explain "kikka")
    (me/humanize {:errors {:re {:error/message "not a number"}}}))
; => ["not a number"]
#2020-10-1313:26dangercoderMy goal is to have an error message like this: "Value: %s does not comply with regexp: %s" 🙂#2020-10-1313:34ikitommimaybe:
(defn regex-error [{:keys [value schema]} _]
  (str "Value: " value " does not comply with regexp: " (first (m/children schema))))

(-> #"\d+"
    (m/explain "kikka")
    (me/humanize {:errors {:re {:error/fn regex-error}}}))
; => ["Value: kikka does not comply with regexp: \\d+"]
#2020-10-1313:40dangercoderI'll check it out. Thanks for the inspiration @ikitommi#2020-10-1309:15raymcdermottI am hitting a dumb problem with references#2020-10-1309:15raymcdermott
(ns data.recursive-case
  (:require [malli.util :as mu]))

(def asset
  [:map
   [:asset/id string?]
   [:asset/status [:enum :initialized :accepted :active :revoked :failed]]
   [:asset/asset {:optional true} asset]])

(def org-asset
  (mu/merge
    asset
    [:map
     [:asset/org :org]]))

(def user-asset
  (mu/merge
    asset
    [:map
     [:asset/user :user]]))
#2020-10-1309:16raymcdermottasset works OK but merge fails#2020-10-1309:16raymcdermott
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema #object[clojure.lang.Var$Unbound 0x4bcbe824 "Unbound: #'data.recursive-case/asset"]}
#2020-10-1309:17raymcdermottpretty sure I've misunderstood something to do with registries so would appreciate if anyone can suggest what's up this#2020-10-1309:27kwrooijenI think this line is wrong in the asset def?
[:asset/asset {:optional true} asset]]
#2020-10-1309:27kwrooijenYou're referring to asset, but it's in the definition itself#2020-10-1309:28kwrooijenJudging from the naming, you actually want it to be recursive?#2020-10-1309:33kwrooijenYou might need to add asset into a registry which you actually want to use it in. e.g. something like this:
{::asset
 [:map
  [:asset/id string?]
  [:asset/status [:enum :initialized :accepted :active :revoked :failed]]
  [:asset/asset {:optional true} [:ref ::asset]]]}
https://github.com/metosin/malli#recursive-schemas
#2020-10-1312:54raymcdermottyes ... I figured it was a registry thing. Thanks for the advice#2020-10-1311:52katox@ikitommi I have been bitten by round-trip java.time.Instant -> string -> malli -> #inst. The problem manifests in java9+ because of increased precision from miliseconds to microseconds. I noticed there is a bigger plan for time handling code but in the meantime I patched the string->date transformer. Should I do a PR for that? See https://github.com/metosin/malli/compare/master...katox:timestamp-frac-precision#2020-10-1313:24miikkaSeems like a good idea (as long as the code still works on Java8 to the extent that it can work)#2020-10-1407:50katoxI tested the change in both j8 and j11. It works the same. Issuing PR.#2020-10-1315:34shemi have JSON data that has entries like "Start":new Date(1413147600000) . clojure JSON parsers (at least Cheshire and Jsonista) choke on this. i wonder if it's possible to use malli.core/decode with the :enter interceptor to parse this and transform it to "Start": 1413147600000 ? my first attempt produces an exception#2020-10-1315:34shem#2020-10-1318:19sparkofreasonSchema inference is awesome going to save me a tone of time. Before I start reinventing a wheel, is anybody working on transforming malli schema to Datomic schema?#2020-10-1516:23ikitommihaven’t seen anything for this, looking forward to your example 🙂#2020-10-1714:24sparkofreasonLooks to be pretty straightforward. I've done this with spec before. There were two headaches there, one being the lack of a public data representation for specs, second being opaque predicates. In the latter case we used test.check to generate examples and tried to infer from that. I think you do the same thing here, but leveraging malli's inference.#2020-10-1515:49ikitommi@raymcdermott noticed that none of the malli.util helpers deref the refernce schemas, wrote an https://github.com/metosin/malli/issues/281. Before that, to use registry references, one needs to deref those manually, one option being:
(require '[malli.core :as m])
(require '[malli.registry :as mr])
(require '[malli.util :as mu])

(mr/set-default-registry!
  (mr/composite-registry
    (m/default-schemas)
    {:asset/id string?
     :asset/status [:enum :initialized :accepted :active :revoked :failed]
     :asset/asset [:map
                   :asset/id
                   :asset/status
                   [:asset/asset {:optional true} [:ref :asset/asset]]]}))

(def org-asset
  (mu/merge
    (m/deref :asset/asset)
    [:map [:asset/org [:= "org"]]]))

org-asset
;[:map
; [:asset/id :asset/id]
; [:asset/status :asset/status]
; [:asset/asset {:optional true} [:ref :asset/asset]]
; [:asset/org [:= "org"]]]

(def user-asset
  (mu/merge
    (m/deref :asset/asset)
    [:map [:asset/user [:= "user"]]]))

user-asset
;[:map
; [:asset/id :asset/id]
; [:asset/status :asset/status]
; [:asset/asset {:optional true} [:ref :asset/asset]]
; [:asset/user [:= "user"]]] 
#2020-10-1515:51ikitommialso, the declarative :merge would allow registries like this:
{:asset/id string?
 :asset/status [:enum :initialized :accepted :active :revoked :failed]
 :asset/asset [:map
               :asset/id
               :asset/status
               [:asset/asset {:optional true} [:ref :asset/asset]]]
 :asset/org-asset [:merge :asset/asset [:map [:asset/org [:= "org"]]]]
 :asset/user-asset [:merge :asset/asset [:map [:asset/user [:= "user"]]]]}
#2020-10-1516:21ikitommi@shem, you example doesn’t seem legit syntax. But for JSON, you need to defin :decode/json key. Something like:
(m/decode
  [:map
   [:Duration string?]
   [:Start {:decode/json (fn [x] (mt/-string->long (last (re-find #"(\d+)" x))))} string?]]
  {:Duration "kikka"
   :Start "new Date(1413147600000)"}
  (mt/json-transformer))
;{:Duration "kikka"
; :Start 1413147600000}
#2020-10-1516:23ikitommiif your new Date is coming from js, it’s a string of Mon Oct 13 2014 00:00:00 GMT+0300 (Eastern European Summer Time) {}, and you need to run it via some date-transforming fn. hope this helps.#2020-10-1516:37ikitommiWould it be a breaking change if m/deref was recursive and would would return the input schema in case it was not a RefSchema? I would like it to be:
(require '[malli.core :as m])

(m/schema [:schema [:schema [:schema [:map [:x int?]]]]])
; => [:schema [:schema [:schema [:map [:x int?]]]]]

(m/deref (m/schema [:schema [:schema [:schema [:map [:x int?]]]]]))
; => [:map [:x int?]]
, instead of:
(m/deref (m/schema [:schema [:schema [:schema [:map [:x int?]]]]]))
; => [:schema [:schema [:map [:x int?]]]]
#2020-10-1516:38ikitommialso:
(m/deref [:map [:x int?]])
; => [:map [:x int?]]
instead of:
(m/deref [:map [:x int?]])
; Execution error (IllegalArgumentException)
; No implementation of method: :-deref of protocol: #'malli.core/RefSchema found for class: malli.core$_map_schema$reify$reify__4587
#2020-10-1517:25eraadHi guys! I’m looking for any pointers on how to transform keys (ie snake to kebab) in nested schemas. I have the following schema:
(def Invoice
  [:map {:registry {::address Address
                    ::tax-line TaxLine}}
   [:issue_date [string? epoch]]
   [:series string?]
   [:sequence [string? {:decode/string mt/-string->long}]]
   [:currency [string? (lookup-ref :currency/code)]]
   [:customer_legal_name string?]
   [:customer_tax_id string?]
   [:customer_tax_id_type [string? (lookup-ref :tax-id/type)]]
   [:customer_email {:optional true} string?]
   [:customer_address {:optional true} ::address]
   [:tax_lines [:vector ::tax-line]]
   [:subtotal [string? {:decode/string mt/-string->double}]]
   [:tax [string? {:decode/string mt/-string->double}]]
   [:total [string? {:decode/string mt/-string->double}]]])
When i do:
(defn coerce-request
  [schema req]
  (if (m/validate schema req)
    (m/decode schema
              req
              nil
              (mt/transformer
               mt/json-transformer
               mt/string-transformer
               {:name :epoch}
               {:name :lookup-ref}
               (mt/key-transformer {:decode k/->kebab-case})))
    (-> schema
        (m/explain req)
        (me/humanize))))
#2020-10-1517:25eraadcustomer_address map keys are not affected#2020-10-1518:06ikitommi@eraad decoding is a process of taking an invalid value and transforming it to a valid (EDN) value. Because of this, the key-transformer runs the transformations on :enter phase. All the top-level map keys are transformed into invalid (kebab)-values and the :customer_address gets a value of :customer-value. After this, the keys don’t match the schema and the child decoders are no-op.#2020-10-1518:08ikitommiif you need to transform values out from the valid (EDN) definitions, you can use m/encode. the key-transformer runs in :leave phase there, in your case as the last thing as it’s also last transformer in the chain.#2020-10-1518:09eraadHi @ikitommi thank you for the response :thumbsup:#2020-10-1518:10ikitommihere’s a minimal sample:
(m/decode
  [:map {:registry {::address [:map [:street string?]]}}
   [:customer_address ::address]]
  {:customer_address {:street "hämeenkatu"}}
  (mt/transformer
    (mt/key-transformer {:decode #(-> % name str/upper-case keyword)})))
; => {:CUSTOMER_ADDRESS {:street "hämeenkatu"}}

(m/encode
  [:map {:registry {::address [:map [:street string?]]}}
   [:customer_address ::address]]
  {:customer_address {:street "hämeenkatu"}}
  (mt/transformer
    (mt/key-transformer {:encode #(-> % name str/upper-case keyword)})))
; => {:CUSTOMER_ADDRESS {:STREET "hämeenkatu"}}
#2020-10-1518:12eraadNice, thanks!#2020-10-1518:13eraadI’m really loving Malli, tried Spec+Spec-tools for a bit but then tried Malli and its great#2020-10-1518:13ikitommiin your case, you can write your own rename-keys like this (running on :leave in decode):
(m/decode
  [:map {:registry {::address [:map [:street string?]]}}
   [:customer_address ::address]]
  {:customer_address {:street "hämeenkatu"}}
  (mt/transformer
    {:decoders {:map {:leave (mt/-transform-map-keys #(-> % name str/upper-case keyword))}}}))
; => {:CUSTOMER_ADDRESS {:STREET "hämeenkatu"}}
#2020-10-1518:15ikitommi“on :decode, for :maps run this function f on :leave”. transformers are quite simple in the end 😉#2020-10-1518:15eraadCoo, I was not being aware of :enter and :leave, that’s the missing part#2020-10-1615:23ikitommiDid a PR on making :schema and :ref schemas behave better: https://github.com/metosin/malli/pull/282. It has a BREAKING CHANGE as m/deref would be recursive and does not throw.#2020-10-1615:26ikitommithese work after the PR: 1. mu/merge and mu/union with references:
(mu/merge
  [:schema [:map [:x int?]]]
  [:map [:y int?]])
; => [:map [:x int?] [:y int?]]
2. mu/subschemas with both :ref & :schemas:
(mu/subschemas
  [:ref {:registry {"Address" [:map
                               [:street :string]
                               [:address [:ref "Address"]]
                               [:neighbor [:ref "Neighbor"]]]
                    "Country" [:map [:name [:= "finland"]]]
                    "Neighbor" [:ref "Address"]}}
   "Address"])
;[{:path [], :in [], :schema [:ref {:registry {"Address" [:map
;                                                         [:street :string]
;                                                         [:address [:ref "Address"]]
;                                                         [:neighbor [:ref "Neighbor"]]]
;                                              "Country" [:map [:name [:= "finland"]]]
;                                              "Neighbor" [:ref "Address"]}}
;                             "Address"]}
; {:path [0 0], :in [], :schema [:map
;                                [:street :string]
;                                [:address [:ref "Address"]]
;                                [:neighbor [:ref "Neighbor"]]]}
; {:path [0 0 :street], :in [:street], :schema :string}
; {:path [0 0 :address], :in [:address], :schema [:ref "Address"]}
; {:path [0 0 :neighbor], :in [:neighbor], :schema [:ref "Neighbor"]}]
#2020-10-1615:28ikitommiif you are in business of generating forms from schemas, mu/subschemas is your best friend.#2020-10-1615:29ikitommim/walk can be configured with ::m/walk-refs and ::m/walk-schema-refs to walk over none, some of all references. Does not StackOverFlow, walks a given reference just once. cheers.#2020-10-1621:02dangercoderis there some kind of way to speed up m/validate? I have a real edge case, I automatically generated a schema for a structure that potentially can contain 1294 fields (a tree structure...) when I ran validate it took a few seconds on a empty map.#2020-10-1621:03dangercoderwould it run quicker if I used a registry for things that are "common"?#2020-10-1621:04borkdudeI think malli supports something like lazy validation and/or lazy schemas - I've seen this being mentioned in a conversation with ikitommi and jeroenvandijk#2020-10-1623:41jeroenvandijk@jarvinenemil not sure if it is still idiomatic Malli and it can be more optimized, but here is how you could do lazy loading (see lookup-type) https://github.com/jeroenvandijk/aws.cloudformation.malli/blob/master/src/adgoji/aws/cloudformation/malli/validation.clj#2020-10-1701:39ikitommi@jarvinenemil are you using m/validator or m/validate? the latter is a convenience fn for one-shot things: if given schema syntax, for every call, it parses it, creates the Schema instance tree and creates a pure validator for it and runs it once and throws it away. m/validator returns that pure validation function of x -> boolean , so if you can keep that, you pay for the parsing & optimizing just once.#2020-10-1701:40ikitommisame applies to m/explain vs m/explainer, m/decode vs m/decoder etc.#2020-10-1701:41ikitommihttps://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1079-L1085#2020-10-1701:45ikitommiat best, given a deeply nested schema with 1k fields, the m/decoder can return a indentity function if there are no decoding functions mapped to any of the fields or structure in general. Can't get much faster than that 😉#2020-10-1709:55dangercoder@ikitommi I did a one off using validate. validator was just what I was looking for, thanks!#2020-10-1714:15ikitommicurious about the perceived perf gains, hopefully many orders of magnitude..#2020-10-1715:02dangercoderUsing validate (cc/quick-bench (m/validate schema {})) Evaluation count : 6 in 6 samples of 1 calls. Execution time mean : 21.971360 sec Execution time std-deviation : 2.848014 sec Execution time lower quantile : 20.065855 sec ( 2.5%) Execution time upper quantile : 26.840086 sec (97.5%) Overhead used : 7.984810 ns Using validator (cc/quick-bench (validator {})) Evaluation count : 31504632 in 6 samples of 5250772 calls. Execution time mean : 12.055681 ns Execution time std-deviation : 0.575152 ns Execution time lower quantile : 11.599548 ns ( 2.5%) Execution time upper quantile : 12.803005 ns (97.5%) Overhead used : 7.984810 ns#2020-10-1715:19ikitommithat's 9 orders of magnitude 😮😮😮 . Are you starting a power plan while creating the schema as it takes 20 seconds???#2020-10-1715:21ikitommianyway, good reason to cache the validator :)#2020-10-1715:21dangercoderI converted a excel-sheet from business people to a malli schema so I can convert the malli schema to a json-schema that can be used from several languages 😄#2020-10-1715:28dangercoderI anonymized the fields and havent added regexp yet but just to give u an idea on how big this thing is. @ikitommi I did not decide this, the real world is sometimes a bit messy 😄.#2020-10-1714:16ikitommiWIP, https://github.com/metosin/malli/pull/283:
(def registry (merge (m/default-schemas) (mu/schemas)))

(def Merged
  (m/schema
    [:merge
     [:map [:x :string]]
     [:map [:y :int]]]
    {:registry registry}))

Merged
;[:merge
; [:map [:x :string]]
; [:map [:y :int]]]

(m/deref Merged)
;[:map 
; [:x :string] 
; [:y :int]]

(m/validate Merged {:x "kikka", :y 6})
; => true
#2020-10-1714:19ikitommijust :merge , :union and :select-keys now, should be easy to add more, with the util helpers. current Schema impls in malli.util:
(defn -merge [] (-util-schema {:type :merge, :f (-reducing merge)}))
(defn -union [] (-util-schema {:type :union, :f (-reducing union)}))
(defn -select-keys [] (-util-schema {:type :select-keys, :min 2, :max 2, :f (-applying select-keys)}))
#2020-10-1717:04jfntn👋 I’m looking for a way to use an accumulator during a schema walk.#2020-10-1717:04jfntnThat’s easy to hack from the outside in with say an atom but I was wondering if there is cleaner way to do this from the inside out instead?#2020-10-1717:05jfntnIt seems like it’d be possible to reify a Walker that expects the walk fn to return [schema context] and merges the returned context into the options to pass to the next step…#2020-10-1717:40ikitommiyes, you should implement a custom Walker. find-first might be a good example for that: https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L37-L51#2020-10-1717:41ikitommior: subschemas: https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L150-L163#2020-10-1718:05jfntnThanks for the pointers!#2020-10-1718:06jfntnI guess doing this without some atom would require some kind of state monad so I think I’ll stick with your examples 😂#2020-10-1718:21ikitommiif you get a generic accumulating walker, might be a good example in https://github.com/metosin/malli/blob/master/docs/tips.md.#2020-10-1914:59Vincent CantinThere might be a way to bridge Specter and Malli for that purpose.#2020-10-1718:45mike_ananev@ikitommi can I have docstring for spec in swagger output?#2020-10-1719:21ikitommi@mike1452 what kind of docstring are you looking for? :title, :description and :default are automatically copied to both JSON Schema and Swagger.#2020-10-1719:23ikitommiJSON Schema references#2020-10-1807:21mike_ananev@ikitommi thanks! this example is what I need.#2020-10-1814:10ikitommiI think declarative schema transformations is done now: https://github.com/metosin/malli/pull/283.#2020-10-1906:09ikitommi🎉#2020-10-1907:17dharriganFantastic! Going to try these out now!#2020-10-1907:48ikitommihttps://malli.io/?value=%7B%3Atype%20%22Cat%22%2C%20%3Aname%20%22Viivi%22%2C%20%3AhuntingSkill%20%3Aadventurous%7D&amp;schema=%5B%3Aschema%20%7B%3Aregistry%20%7B%22Pet%22%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atype%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20string%3F%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Cat%22%20%5B%3Amerge%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Pet%22%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atype%20%5B%3A%3D%20%22Cat%22%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3AhuntingSkill%20%5B%3Aenum%20%7B%3Adescription%20%22The%20measured%20skill%20for%20hunting%22%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Aclueless%2C%20%3Alazy%2C%20%3Aadventurous%2C%20%3Aaggressive%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Dog%22%20%5B%3Amerge%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%22Pet%22%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atype%20%5B%3A%3D%20%22Dog%22%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3ApackSize%20%5B%3Aint%20%7B%3Amin%200%2C%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Adefault%200%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Adescription%20%22the%20size%20of%20the%20pack%20the%20dog%20is%20from%22%7D%5D%5D%5D%5D%7D%7D%0A%20%5B%3Amulti%20%7B%3Adispatch%20%3Atype%7D%20%22Cat%22%20%22Dog%22%5D%5D#2020-10-1913:32ikitommiwhat a brilliant/horrible idea: add optional :object schema, which allows maps to be described using clojure maps. Kinda like data-specs but for malli:
[:object
 {"name" [:required :string]
  "age" [:required :number :positive :integer]
  "email" [:string :email]
  "website" [:string :url]
  "createdOn" [[:date {:default "2020-18-10"}]]
  "address" [:object
             {"street" [[:string {:min 1}]]
              "zip" [:int]}]}]
#2020-10-1913:35ikitommior:
[:object
 {"name" [:and :required :string]
  "age" [:and :required :number :positive :integer]
  "email" [:and :string :email]
  "website" [:and :string :url]
  "createdOn" [:date {:default "2020-18-10"}]
  "address" {:street [:string {:min 1}]
             :zip :int}}]
#2020-10-1914:17borkdudea bit like Schema, but more EDN-like#2020-10-1914:37ikitommi☝️. simple formats like Schema and data-spec are kinda easy, but not simple: the core utilties (`select-keys`, assoc etc.) almost work, unless you have a wrapper for keys like ds/opt or s/optional-key in case they don’t. Also, value wrappers are needed to add visible properties/meta-data to schemas. And there is no order for keys. But, super nice for many things like defining inlined route parameters:
["/plus"
 {:get {:summary "plus with spec query parameters"
        :parameters {:query {:x int?, :y int?}}
        :responses {200 {:body {:total int?}}}
        :handler handle-plus}]
#2020-10-1914:40ikitommiif there was a litemalli coercion in reitit, one could swap the data-spec apps almost 1:1 to it.#2020-10-1914:47pithylessI prefer #1 to #2; the latter is making nested map notation implicit and special for :object and I don't know if it's worth it.#2020-10-1914:51pithylessThat reitit example could just as easily be solved by a utility function that takes the concise nested map notation and converts it to a tree of :object#2020-10-1914:52borkdudesomeone should write malli-tools ;)#2020-10-1914:52ikitommi:grinning_face_with_one_large_and_one_small_eye:#2020-10-1914:58pithylessDoes :object offer different semantics or tradeoffs than :map? Or would it deref/merge down to a :map schema? Or would we have two things that are the same but different? :]#2020-10-1914:59pithylessTo be clear, I prefer writing the map notation and was always confused why :map used vectors; but that ship has sailed... or has it?#2020-10-1915:01borkdudewe're in a dynlang, we can do everything we want ;)#2020-10-1915:03pithylessI'm thinking :object could be just a nice alternative syntax that would "compile down" to a :map. But maybe I'm just missing the point. :)#2020-10-1915:04ikitommihttps://github.com/metosin/malli/issues/286#2020-10-1915:06borkdudewhy it is called object though?#2020-10-1915:06borkdudelike JavaScript object?#2020-10-1915:08borkdudeI was thinking like: wow, is this a schema to describe Java objects like bean-stuff?#2020-10-1915:08borkdudeanyone not working on front-ends might think the same#2020-10-1915:09ikitommivector syntax for :map allows simple way to define properties for it and retains the order of keys. But yes, we can add more schemas to build things in different ways. Could have also optional :`json-schema` that allows any JSON Schema in it etc.#2020-10-1915:13ikitommithe name could be anything, just flushing stuff from my brain (after a vacation).#2020-10-1915:19pithylessIMO, it would be confusing to see both :map and :object somewhere in my system output and they were semantically the same thing (a representation of known key-value mapping); so the way I see it: 1. if we want an alternative writing notation, why not just make it a utility function? (instead of introducing a new kind of schema) 2. if we want an alternative notation that is a data-literal (utility functions need runtime support), then we could introduce an alternative name (like :object ); but it could just as easily be :kv or :map2 ; or even why not extend existing :map to support both kinds of notations (vectors or maps) 3. if this is something similar to :map but offers possibly different semantics, than currently those semantics aren't clear to me; otherwise it'd be great if we didn't have two things that represent essentially the same thing (this would make merging, etc. more complicated to deal with later on)#2020-10-1915:21pithyless(This is also why I like having :map and :map-of , because they represent different semantics; even though they are using the same internal datastructure)#2020-10-1915:38ikitommiRelated to 2, I think they should be kept separate (if implemented). If :map supported both syntax, there would be ambiguity issues:
[:map
 {:id :int
  :title :string}]
is that empty map with props? or map with no props but with keys?
#2020-10-1915:43ikitommiIf I would implement that today, it would be a separate ns, malli.object etc, with a Schema impl, a helper and a extra registry snipplet so one can merge the schemas into a registry easily:
(require '[malli.core :as m])
(require '[malli.object :as mo])

(def registry (merge (m/default-schemas) (mo/schemas)))

(m/schema
  [:object 
   {:id int?
    :address [:object {:street string?}]}]
  {:registry registry})
#2020-10-2112:28salva-xfHello, I found malli a couple of days ago and I'm liking it although I don't know if I really understand it well, is malli suitable for defining state machines for example? In general, it is suitable for defining data structures to go through and generate code? generate the models for the orientdb and the datascript for example? Would it be suitable for, for example, defining IFML structures and generating the user interfaces from them? These are the questions that I try to answer while I see the documentation, thank you very much#2020-10-2116:19Lucy WangMalli could do most what spec could do, data validation, data transform/coercion, data generation etc. This might be a good refererence https://www.pixelated-noise.com/blog/2020/09/10/what-spec-is/#2020-10-2119:36ikitommiwe have done separate definitions for state machines, as pure data too. signals to the state machine have (input) schemas, guards, actions and arbitrary other control data to them and the uis are derived from the combination of fme + schema definitions: form can be derived from schemas, available actions (and popovers why they are not available) from guards and the available actions in from the fme step definitions. lot’s of ui-stuff like extra re-frame action definitions, button texts and model guidees in the process definitions too.#2020-10-2119:39ikitommimost real uis need more that “automatically derived from schema”, but, it seems good for many cases like erp’ish internal stuff. Finland is the land of forms, great to have some extra leverage to build those ;)#2020-10-2119:41ikitommigreat talk by @U050CBXUZ & @U0DJK1VH6 on building declarative uis with clj: https://www.youtube.com/watch?v=IekPZpfbdaI#2020-10-2210:39borkdude@ikitommi I was noticing a regex/parse function in one of the examples. Why is there a special regex/parse function: shouldn't all schema's support parse - not only regex schemas? And maybe parse is just a special case of a transformer?#2020-10-2210:41borkdudeNote: this question is coming from a relatively inexperienced user of malli#2020-10-2210:51ikitommi@borkdude good question! it’s the internal api, not yet integrated into malli.core. after it’s integrated, there will ne a`-destructure` or -parse method in Schema. My first guess was that there is no need to have a reverse function for that in Schema (like spec has conform and unform), but not 100% sure.#2020-10-2210:54ikitommioh, haven’t pushed the current draft into github. will have time tomorrow for this, will push the stuff out.#2020-10-2509:49borkdudehttps://www.reddit.com/r/Clojure/comments/jhq899/did_anyone_migrate_a_nontrivial_project_from_spec/#2020-10-2510:14jeroenvandijkI did, yes it is more involved 🙂#2020-10-2510:17jeroenvandijkTo be more specific, this project is a (conceptual) port from a clojure.spec project https://github.com/jeroenvandijk/aws.cloudformation.malli#2020-10-2510:18jeroenvandijkI was generating the clojure.spec definitions from the AWS Cloudformation spec. This generation part was much easier with Malli
#2020-10-2510:19jeroenvandijkThere are enough differences that I cannot consider it as a 1 on 1 port#2020-10-2510:40borkdudeFeel free to respond on Reddit as this was not my own question, just forwarded it here#2020-10-2510:51jeroenvandijkyeah thanks. I’m not using reddit. I hope the reddit user finds it here 😅#2020-10-2516:28borkdudeIs there any guidance on how to write your own transformer?#2020-10-2516:35ikitommi@borkdude there is no guide yet, but lot of tests and some samples in the README.#2020-10-2516:36ikitommihappy to help if you need any.#2020-10-2516:38borkdude@ikitommi I think I asked this before, but I can't find any docs on this, nor can I find the conversation on Zulip. So here it goes:
(def Schema
  [:map [:x int?]])

(defrecord Wrapper [obj loc])

parsed 
;;=> {#script.Wrapper{:obj :x, :loc {:row 1, :col 2, :end-row 1, :end-col 4}} #script.Wrapper{:obj 1, :loc {:row 1, :col 5, :end-row 1, :end-col 6}}}

(defn wrapper-transformer []
  (mt/transformer
   {:name :wrapper
    :default-decoder :obj
    :default-encoder (fn [obj]
                       (->Wrapper obj nil))}))

(prn (m/decode Schema parsed wrapper-transformer)) ;;=> nil ... ???
#2020-10-2516:40borkdudeso I want to validate/transform the wrapped value according to the schema#2020-10-2516:41borkdudeany value that is not Wrapped should just be transformed as is#2020-10-2516:41borkdudebut a value that is wrapped should be unpacked using :obj#2020-10-2516:42borkdudeand I want to give an error message based on :loc#2020-10-2516:44ikitommioh, i recall. need to dig in my histories too if there was/is a good answer.#2020-10-2516:45borkdudeI tried this:
(defn unwrap [x]
  (if (instance? Wrapper x)
    (:obj x)
    x))

(defn wrapper-transformer []
  (mt/transformer
   {:name :wrapper
    :default-decoder unwrap
    :default-encoder (fn [obj]
                       (->Wrapper obj nil))}))
but this just returns the entire object itself
#2020-10-2516:47borkdudeit seems it doesn't handle the value recursively#2020-10-2516:50ikitommithe problem is that the decoder is first given a map, it calls :obj on it, which return nil.#2020-10-2516:51ikitommiI would use clojure.walk/pre-walk for first sweep of decoding.#2020-10-2516:52ikitommithe decoding here walks: 1. the :map 2. the keys (`:x`)#2020-10-2516:52ikitommiunless the Wrappedis already decoded on 1, there is no :x and the decoding stops.#2020-10-2516:53ikitommiso, with malli, you would need to decode all the keys & values on :map step also. doable, but extra noise 😞#2020-10-2516:55ikitommi
(ns user
  (:require [malli.core :as m]
            [malli.transform :as mt]))

(def Schema
  [:map [:x int?]])

(defrecord Wrapper [obj loc])

(def parsed {(map->Wrapper {:obj :x, :loc {:row 1, :col 2, :end-row 1, :end-col 4}})
             (map->Wrapper {:obj 1, :loc {:row 1, :col 5, :end-row 1, :end-col 6}})})

(defn unwrap [x]
  (if (instance? Wrapper x) (:obj x) x))

(defn wrapper-transformer []
  (mt/transformer
    {:name :wrapper
     :decoders {:map (fn [x] (reduce-kv (fn [acc k v] (assoc acc (unwrap k) (unwrap v))) {} x))}
     :default-decoder unwrap
     :default-encoder (fn [obj] (->Wrapper obj nil))}))

(m/decode Schema parsed wrapper-transformer) ;;=> nil ... ???
; => {:x 1}
#2020-10-2516:55ikitommibut, this is much simpler:
(clojure.walk/prewalk unwrap parsed)
; => {:x 1}
#2020-10-2516:57borkdudeYes, but how do I get validation errors if I first call prewalk on this thing?#2020-10-2516:58borkdudebased on :loc#2020-10-2516:58ikitommiI’ll check more of this later.#2020-10-2517:02borkdudeThanks. Me too, cooking dinner :)#2020-10-2517:07borkdudeHmm, in spec I would maybe have to spec a key as ::wrapped-int and then coerce it after it was checked?#2020-10-2517:13borkdudewhich is not ideal#2020-10-2517:25ikitommican the wrapped by anywhere? Wrapped map/vector/set? All values are wrapped (any nested edn value)? Or just keys and values in the map?#2020-10-2517:26ikitommiI have an idea, but would like to understand the case first.#2020-10-2517:54borkdude@ikitommi The use case is preserving location information for non-iobjs and using that for error messages while validating malli schemas#2020-10-2517:54borkdudee.g. you want to validate an EDN file and you get an error: this should be an int, on line 5, row 12#2020-10-2518:16borkdude@ikitommi This is the complete code: deps.edn:
{:deps {metosin/malli {:mvn/version "0.2.1"}
        borkdude/edamame {:git/url ""
                          :sha "ba93fcfca1a0fff1f68d5137b98606b82797a17a"}}}
(ns script
  (:require [edamame.core :as e]
            [malli.core :as m]
            [malli.transform :as mt]))

(def Schema
  [:map [:x int?]])

(defrecord Wrapper [obj loc])

(defn iobj? [x]
  (instance? clojure.lang.IObj x))

(def parsed
  (e/parse-string "{:x 1}"
                  {:postprocess
                   (fn [{:keys [:obj :loc]}]
                     (if (iobj? obj)
                       (vary-meta obj merge loc)
                       (->Wrapper obj loc)))}))

(defn unwrap [x]
  (if (instance? Wrapper x)
    (:obj x)
    x))

(defn wrapper-transformer []
  (mt/transformer
   {:name :wrapper
    :default-decoder unwrap
    :default-encoder (fn [obj]
                       (->Wrapper obj nil))}))

;; (prn parsed)
(prn (m/decode Schema parsed wrapper-transformer))
#2020-10-2519:35borkdude@ikitommi Found the conversation here: https://clojurians.zulipchat.com/#narrow/stream/180378-slack-archive/topic/malli/near/206235546#2020-10-2519:52borkdude@ikitommi So the way it can work is:
(defn fail! [schema {:keys [:obj :loc]}]
  (throw (ex-info (str obj " did not satisfy " schema
                       " [at " (str (:col loc)":"(:row loc)) "]") {})))

(def <42 [:and 'int? [:< 42]])

(defn lift-non-iobj-schema [schema]
  [:map {:encode/success :obj,
         :encode/failure (partial fail! schema)} [:obj schema]])

(def Schema [:map [:a (lift-non-iobj-schema <42)]])

(def valid? (m/validator Schema))
(def success (m/encoder Schema (mt/transformer {:name :success})))
(def failure (m/encoder Schema (mt/transformer {:name :failure})))
#2020-10-2519:52borkdudeE.g. this will print:
(prn (parse-validate-and-transform "{:a 42}"))
#2020-10-2519:52borkdude
Syntax error (ExceptionInfo) compiling at (script.clj:32:1).
42 did not satisfy [:and int? [:< 42]] [at 5:1]
#2020-10-2519:53borkdudeThe downsize of this is that I have to wrap schemas myself in case I want to check something non-iobj-ish (strings, keywords, numbers)#2020-10-2519:57borkdudeAnd this doesn't seem to work for keywords for example:
(def <42 [:and 'int? [:< 42]])

(defn lift-non-iobj-schema [schema]
  [:map {:encode/success :obj,
         :encode/failure (partial fail! schema)}
   [:obj schema]])

(def Schema [:map [(lift-non-iobj-schema :a) (lift-non-iobj-schema <42)]])
#2020-10-2519:58borkdudeanyway, maybe this is a too niche use case#2020-10-2520:03ikitommihave an idea, need 30min to test & play#2020-10-2520:05borkdudetake yer time#2020-10-2520:06borkdudeif this can be fixed, potentially it would also work for rewrite-cljc structures: normal malli specs, but they run over the rewrite-cljc nodes using a simple function that looks at the actual value#2020-10-2520:07borkdudeit's basically a projection from wrapped thing to the essential value while the wrapped thing has more info to produce useful output in case of failure to parse/validate#2020-10-2610:16ikitommi@jarvinenemil the huge schema creation perf issue: culprit is satisfies?, which can be removed using a special extending protocols to protocols pattern. With quick test it’s 22sec -> 2sec. With some tuning, 0.8sec. Fix is on the backlog.#2020-10-2610:17dangercoderawesome @ikitommi, good find!#2020-10-2621:26mruzekwHi all, what are the options currently to use Malli like you can with clojure.spec and ghostwheel or orchestra? So far I've found teknql/aave, but I wondered if there were others#2020-10-2623:04rschmukler@mruzekw I think aave is the only one right now. It's definitely been on my backburner though so happy to accept PRs if you feel like hacking on it. Otherwise I wouldn't be surprised if ikitommi ends up releasing his own thing. Happy to update aave to 0.2.0 if that's what is keeping you back from it. (No pressure to use it either!)#2020-10-2716:43lmergenjust a shout-out that i'm biting the bullet and migrating our project with 4k+ LoC full of specs to malli and i'm totally loving it -- the fact that it's just plain symbols + data makes thing so much easier to reason about. thanks a lot for this monumental effort of making malli work!#2020-10-2716:48lmergenjust a sanity check, but it seems to me that using "plain data" for all the schemas (i.e. not wrapping them in m/schema), and then at a later point use a compile-time m/validator and m/explainer is a decent approach, right ? i've just whipped up this macro to perform assertions, which allocates the explainer at compile time, but does the actual validations at runtime
(defmacro assert
  [s v]
  (when *assert*
    (let [explainer# `(m/explainer ~s)]
      `(when-let [ed# (~explainer# ~v)]
         (throw (ex-info "Validation failed"
                         (merge {::v ~v}
                                (me/humanize ed#))))))))
#2022-09-2300:52marciolHey, @lmergen thanks for this snippet. I wonder if this can be part of the default malli distribution @U055NJ5CC#2022-09-2313:24ikitommiI think that m/assert would be good, but it should work in all situations, so I believe the schema creation should happen at runtime.#2020-10-2716:51borkdude@lmergen I'm curious about your experience report. There is a thread on Reddit about exactly this. Feel free to comment there if you want.#2020-10-2716:51borkdude(https://www.reddit.com/r/Clojure/comments/jhq899/did_anyone_migrate_a_nontrivial_project_from_spec/)#2020-10-2716:52rschmukler@lmergen that's exactly what I do. Most (all?) of the malli public API will coerce to a compiled schema, which lets you keep your schema definitions nice and focused. Then compile-time m/validator and m/explainer. Also there are the short hands of m/explain (and m/validate) which create a temporary explainer for you. This isn't performant, but, for use-cases like your macro, it saves you having to create it yourself.#2020-10-2716:53rschmuklerI suppose a slight difference is that your macro will create the explainer at compile time, so up to you if you want that difference!#2020-10-2716:53lmergen@borkdude i started that thread 🙂#2020-10-2716:54borkdudeaaah! :)#2020-10-2716:57lmergenbut yes so far so good -- of course there's a "second system" benefit here, because i have a much better understanding of the model and taking the opportunity to refactor a few things. one of the big changes is that rather than one humongous specs.cljc, i'm now creating many more namespaces, and each of the malli schemas live within those namespaces. the reason for this is that spec works with (namespaced) keywords, so you can get away with putting them in a single namespace, where malli seems to map more natural if you split things up in different namespaces.#2020-10-2717:04rschmukler@lmergen that was exactly my experience as well! One benefit that you get is if you make use of def or defn to return those data schemas, you get the added benefit of having the compiler help check that those things still exist. In spec you can delete / mistype a keyword and get bit, but here, you can get the compiler (or clj-kondo) to help you#2020-10-2810:12borkdude@ikitommi I think I would have a real good use of sequence schemas. Spec is painful to work for command line usage with due to macros, malli is a pleasure because it's data and also compatible with Graal. Context: https://gist.github.com/borkdude/a391146ad81a06c28fb97ccdc1f64d44#2020-10-2810:41jeroenvandijkI was thinking about this too. To find duplicated or similar code etc#2020-10-2810:13borkdudeImagine that the user can specify a schema on the command line to search for code patterns.#2020-10-3000:05eraadHi! I’m using Malli schemas to generate a Swagger spec using Reitit (ie. “issue_date”). These schemas have encoding fns so “issue_date” is coerced to :document/issue-date so it can be saved into Datomic. I want to confirm if the Reitit coercers do validation after coercion right? Because I’m not getting it to validate as “issue_date” but :document/issue-date#2020-10-3000:07eraadIt is what I can see from the code and from the behaviour. If I want to keep the json-like schemas for Swagger generation AND validate using those, I need to do coercion after all that manually right?#2020-10-3000:08eraadOr of course write my own Malli coercer that always decodes.#2020-10-3000:12eraadI would appreciate any pointers on this#2020-10-3016:01ikitommi@lmergen @rschmukler should there be m/assert?#2020-11-0108:59lmergeni think there should be -- i have two functions myself actually, assert and have, where have is inspired by ptaoussanis' truss -- https://github.com/ptaoussanis/truss effectively, have does the assertion + also returns the input value if no failure was detected. this allows for a pattern such as
(let [foo (m/have foo-schema (get x :foo))]
  ...)
it's very effective. i would be happy to send a PR for the assert and/or have implementations i have right now (which seem to be doing the right thing)
#2020-10-3016:02ikitommi@eraad coercion is decode + validate in reitit-malli#2020-10-3016:03ikitommifor responses, I recall it's validate + encode#2020-10-3016:04ikitommihave had a super busy two weeks, should have few full days of malli after next wed.#2020-10-3016:21eraadThanks @ikitommi got you#2020-10-3016:30ikitommithe m/defn, currently have a separate malli.schema ns, and few helper ns's under it. Will have the 1:1 Plumatic Schema m/defn for Malli. Once done, will do the real full malli->clj-kondo integration, that can be used with both malli-defn and aave, and a standard fdef -kinda way to annotate existing functions like in spec.#2020-10-3016:31borkdude@ikitommi exciting!#2020-10-3016:32ikitommimight be worth trying to write spec->malli conversion, so we could reuse all the existing fdefs out there. Also, zillion little things to do :)#2020-10-3019:55rschmukler@ikitommi once we have that I may take a crack at some nrepl middleware and cider integration. Right now you can spec generate and see spec definitions in documentation lookup in emacs. Only thing I miss from spec.#2020-10-3019:56rschmuklerRe: m/assert - why not? I think it'll be attractive for some of those migrating from spec#2020-10-3100:24hoppyjust for funzies, this warning is coming out of shadow-cljs in a node-js release build. I'll be happy to chase it if somebody wants me to. Oh, and p.s., malli running on AM335x embedded device, in case somebody is keeping score.#2020-11-0115:26ikitommi@hoppy https://github.com/metosin/malli/issues/87 help welcome!#2020-11-0118:29lmergenis it correct that (at least currently), transforming enums from json does not yet correctly transform them to keywords if necessary ?#2020-11-0118:54borkdude@lmergen Should enums always be keywords?#2020-11-0118:57lmergenwell I wouldn’t say always — but at least I would expect it to convert it if that leads to a valid enum value#2020-11-0118:58borkduderight#2020-11-0119:16ikitommi@lmergen right. There is no type inferrer with enums. They could be anything [:enum 1 "2" :3 '4]. If you know the type, you can hint it: [:and keyword? [:enum :a :b :c]] and they transform correctly#2020-11-0119:16lmergenoh that works well enough for now #2020-11-0119:19ikitommithe generated JSON Schema with :and is bit off, it creates :allOf out of that. Should merge the results instead if there is just one district :type found.#2020-11-0119:20ikitommi(correct, but not practical)#2020-11-0211:29lmergenis this a valid instant? it can be represented by Instant, but it appears to not be parsable by malli.transform:
(pr-str (mt/-string->date "0000-12-31T23:59:59.999Z"))
=> "0000-12-31T23:59:59.999Z"
#2020-11-0211:31lmergen
1. Caused by java.time.DateTimeException
   Invalid value for YearOfEra (valid values 1 -
   999999999/1000000000): 0
apparently that's the issue
#2020-11-0211:33lmergenseems like a inconsistency to me to what Instant can represent and DateTime can parse#2020-11-0220:40ikitommi@lmergen inst? should be companioned with set of date-types, the issue is here: https://github.com/metosin/malli/issues/49. That said, we just merged an enhancement to inst? transformation, happy to take another one, if there is something still missing. See https://github.com/metosin/malli/pull/280#2020-11-0220:41lmergenI’ll take a look, thanks! This is a corner case anyway#2020-11-0220:42lmergenlooks like this may indeed fix my issue as it’s the same instant -> date -> instant round trip, I’ll give it a try #2020-11-0311:32Markus StrHi, I'm getting started with malli and this migh be a noob question, but why is this an invalid schema error? (I took the source of string? in cljs and changed it to str2? Weirdly string? is valid and the alias isn't
(defn ^boolean str2?
  "Returns true if x is a JavaScript string."
  [x]
  (goog/isString x))
(def Test
  [:map
   [:url str2?]
   [:det map?]
   ]
  )

(-> Test
    (m/explain {:url "sa" :det {:a 5}})
    (me/humanize)
    )
#2020-11-0311:34Markus StrWhy can't I define a boolean predicate function there?#2020-11-0311:41borkdude@US22FPMPX Any reason you're not using string?#2020-11-0311:44Lucy Wang@US22FPMPX try [:fn str2]#2020-11-0311:51ikitommithe default shemas are listed here: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1223-L1269#2020-11-0311:53borkdudeFWIW string? on CLJS is:
(defn ^boolean string?
  "Returns true if x is a JavaScript string."
  [x]
  (goog/isString x))
#2020-11-0311:53Markus Strthanks for the quick repsonse @U04V15CAJ, str2? was just for figuring out if my function was the issue#2020-11-0311:54borkdudeah ok#2020-11-0311:54Markus Str@UP90Q48J3 thanks that works actually, but error messages are not defined then#2020-11-0311:54Markus Strexample:
(defn twitter-url? [s] (re-find #"http[s]?://twitter.*\d+" s))
(def Test
  [:map
   [:url [:fn twitter-url?]]
   [:det map?]
   ]
  )
(-> Test
    (m/explain {:url "" :det {:a 5}})
    (me/humanize)
    )
#2020-11-0311:55Markus Str
(-> Test
    (m/explain {:url "" :det {:a 5}})
    (me/humanize)
    )
#2020-11-0311:55Markus Str{:url ["unknown error"]} then#2020-11-0311:55borkdude@US22FPMPX Have you tried https://github.com/metosin/malli#custom-error-messages?#2020-11-0311:56borkdudeMaybe malli expects the function to return a boolean instead of nil? don't know#2020-11-0311:58Markus StrThanks for the pointer#2020-11-0311:59ikitomminils should be ok, also regexs:
[:map
 [:x #"http[s]?://twitter.*\d+"]
 [:y [:fn {:error/message "invalid"} (constantly false)]]
#2020-11-0312:08Markus StrAh interesting, so regex work, but I can't have something like:
[:x #(re-find #"http[s]?://twitter.*\d+" %)]
Probably have to add to the default schema registry then, to have it work naturally like, [:x twitter-url?] I guess?
#2020-11-0312:10Markus StrI'm going to read the docs for the third time; seems my mental model is not there yet!#2020-11-0312:13Markus Str
(defn twitter-url? [s] (re-find #"http[s]?://twitter.*\d+" s))
(def -twitter-url? [:fn {:error/message "invalid twitter url"} twitter-url?])

(def Test
  [:map
   [:url -twitter-url?]
   [:det map?]
   ]
  )
(-> Test
    (m/explain {:url "" :det {:a 5}})
    (me/humanize)
    )
So this is how I could do it; only backdraw is that I have two functions now; one for normal use, the other specifically for validation
#2020-11-0312:16ikitommicould make the predicate schemas implement IFn, so you could:
(def twitter-url?
  (m/schema 
    [:re {:error/message "invalid twitter url"}
       #"http[s]?://twitter.*\d+"]))

(twitter-url? "")
; => true

(m/validat twitter-url? "")
; => true
#2020-11-0312:28Markus StrLooks promising. For me personally (and I might be totally off) it could be nice for malli to also accept predicate functions directly inside the spec-vector
(defn twitter-url? [s] (re-find #"http[s]?://twitter.*\d+" s))
(def Test
  [:map
   [:url twitter-url?]
   ]
  )
For now I'll read up on the registry and how to add it there
#2020-11-0312:34borkdudeWhat's the reason it requires [:fn ..] right now @U055NJ5CC?#2020-11-0316:21ikitommicould add an option to allow that shortcut (as there is regexs), but in short: it’s too easy to write schemas which do not serialize.#2020-11-0316:22ikitommibut if one is not looking for serialization, it would be easier for sure.#2020-11-0316:22borkdudegood point. maybe for some people serializing a schema is not important?#2020-11-0316:22borkdudejinx#2020-11-0316:23ikitommineed to revisit these for 1.0.0. but will add optional support for plain functions.#2020-11-0316:24ikitommialso, the default registry being immutable. it’s currently easy to use immutable registries, but hard to get a global mutable registry. both could be easy.#2020-11-0316:26borkdudeDo you think malli will be able to do something like grasp?#2020-11-0316:27borkdudeas in, support the things that spec does to match s-expressions#2020-11-0316:27ikitommiI believe so, as soon as there are the sequence schemas.#2020-11-0412:13Lucy Wang> good point. maybe for some people serializing a schema is not important? to be frank I think quite some (if not most) people don't need that ...#2020-11-0313:13lmergenis there something off with generators and the [:and] clause ?
(def schema [:and map?
             [:map
              [:foo string?]]])

(mg/generate schema)
throws an error ("Couldn't satisfy such-that predicate after 100 tries.") this, however, works:
(def schema [:and
             [:map
              [:foo string?]]
             map?])

(mg/generate schema)
my hunch is that it's unable to "deduce" that it should first try to generate the :map , and starts by generating a completely random map which of course never qualifies the [:map [:foo string?]] schema?
#2020-11-0313:17ikitommiyes, the order matters#2020-11-0313:31lmergenstill seems like there's something off with :and :thinking_face:
(def base [:map
           [:foo [:enum :a :b]]])
(def x [:and base
        [:multi {:dispatch :foo}
         [:a [:map [:i string?]]]
         [:b [:map [:j pos-int?]]]]])
seems like mg/generate is unable to generate values for this schema as well
#2020-11-0314:25ikitommi@lmergen
(defmethod -schema-generator :and [schema options] (gen/such-that (m/validator schema options) (-> schema (m/children options) first (generator options)) 100))
#2020-11-0314:26ikitommi^:-- the first one is used for the generator.#2020-11-0314:26lmergenaha!#2020-11-0314:26lmergenbut that means there isn't really a way for me to work around it except writing a custom generator, right ?#2020-11-0314:28lmergen(which is what i just did)#2020-11-0314:29ikitommiyou could merge the base schemas, but merge might not work with :multi. but, there are options, like:
(mg/generate
  [:and
   [:multi {:dispatch :foo}
    [:a [:merge ::base [:map [:i string?]]]]
    [:b [:merge ::base [:map [:j pos-int?]]]]]]
  {:registry (merge
               (m/default-schemas)
               (mu/schemas)
               {::base [:map [:foo [:enum :a :b]]]})})
#2020-11-0314:30lmergeni see#2020-11-0314:31ikitommimerge doesn’t work with :multi currently, this would be best way to do it (if it worked):
(mu/merge
  [:map [:foo [:enum :a :b]]]
  [:multi {:dispatch :foo}
   [:a [:map [:i string?]]]
   [:b [:map [:j pos-int?]]]])
;[:multi {:dispatch :foo}
; [:a [:map [:i string?]]]
; [:b [:map [:j pos-int?]]]]
#2020-11-0314:32lmergeni just wrote a helper function which just does this:
(defn merged
  "Given a list of generators, returns a generator that applies `merge`
  to all the generator results."
  [& gens]
  (gen/fmap (fn [args]
              (reduce merge {} args))
            (apply gen/tuple gens)))

(def schema [:and {:gen/gen (merged (mg/geneator a) (mg/generator b)} a b])
#2020-11-0314:32ikitommi:multi should do type-inferring, so that all map-like :multis act like a map.#2020-11-0314:32lmergenyes#2020-11-0314:32lmergenthat's what i was going to say#2020-11-0314:32lmergen:multi is pretty much always a map anyway#2020-11-0314:34ikitommiunless it’s a tuple / sequence:
(m/validate
  [:multi {:dispatch 'first}
   [:sized [:tuple keyword? [:map [:size int?]]]]
   [:human [:tuple keyword? [:map [:name string?] [:address [:map [:country keyword?]]]]]]]
  [:human {:name "seppo", :address {:country :sweden}}])
; true
#2020-11-0314:34lmergenright#2020-11-0314:35ikitommibut, it’s easy to figure out if it looks like a map and could be merged.#2020-11-0319:02Maciej FalskiHi, I’ve just started with malli, and I’m spiking converting our spec definitions to malli-based. Among others, we’ve defined specs with generators and json decoding for java-time (aka java8) types , like instant, duration , and interval. It’s easy with spec-tools. Now I’m trying to do the same with malli and this is what I came up with:
(def Instant
  (m/-simple-schema
    {:pred            t/instant?
     :type-properties {:error/message "should be an instant"
                       :decode/json   t/instant
                       :encode/json   str
                       :gen/gen       (gen/fmap (fn [[d h m s]] (t/plus (t/instant) (t/days d) (t/hours h) (t/minutes m) (t/seconds s)))
                                                (gen/tuple gen/int gen/int gen/int gen/int))}}))
Does it makes sense? Would there be a better way? I’m aware of this https://github.com/metosin/malli/issues/49, but it’s still open.
#2020-11-0406:13ikitommi@maciej.falski looks good! few small things: • if you want to support also string->edn & edn->string, you should add :decode/string and :encode/string keys. Needed if you use the dates in header/path/yaml/query-params • you could add :type key, .e.g. 'Instant so that the schema forms render correctly • with :type, you could also add the new schmas into registry and they are available for de/serialization too.#2020-11-0406:16ikitommioh, and 'going to talk about mallin in London Clojurians in 8.12. https://www.meetup.com/London-Clojurians/events/274367598/#2020-11-0508:51ikitommiback to function schemas. basics kinda work without varargs, as it requires sequence schemas. todo: emit clj-kondo annotations for real.#2020-11-0508:52ikitommi#2020-11-0508:54ikitommialso todo: pretty errors, e.g. https://github.com/metosin/virhe#2020-11-0906:47ikitommi@borkdude tested the malli + edamame with location preserving metadata. Few notes: • current impl of transform is not aware of the path where the transformation happens. If it did, the solution would be almost trivial • good thing is that we have other tools to go around the limitation: we can walk the schema and inject :in to all subschema properties and use :compile in a transformer to read this info at decoder creation time and to collect all value path -> loc into a request-scoped atom as we decode the Wrapped values • transformers compose, so we can also do string->edn etc in a single sweep • for the transformed value, we call explain and in case of errors, attach the locs from the atom, as explain errors know already both schema and value paths • solution has lot of boilerplate, and found a :set explain bug (wrote https://github.com/metosin/malli/issues/294 ), with that, this could even work 😎https://gist.github.com/ikitommi/e3229a0bcef532d1fa032321713227d3#2020-11-0906:48ikitommigood thing about using a lookup-table is that it doesn’t need to throw and reports all errors.#2020-11-0906:51ikitommialso, if edamame could accumulate and provide the full path to a given element in :postprocess, it could be used here too, e.g. extra key + value in Wrapped like :in [:tags "address"].#2020-11-0906:53ikitommi
(def Address
  [:map
   [:id string?]
   [:tags [:set keyword?]]
   [:address
    [:map
     [:street string?]
     [:city string?]
     [:zip int?]
     [:lonlat [:tuple double? double?]]]]])

;; string->edn, no coercion
(let [coerce (coercer Address)]
  (coerce (slurp "schema.edn")))
;{:schema [:map
;          [:id string?]
;          [:tags [:set keyword?]]
;          [:address
;           [:map
;            [:street string?]
;            [:city string?]
;            [:zip int?]
;            [:lonlat [:tuple double? double?]]]]],
; :value {:id "Lillan",
;         :tags #{":hotel" :coffee :artesan},
;         :address {:lonlat [61.4858322 23.7854658],
;                   :city "Tampere",
;                   :street "Ahlmanintie 29",
;                   :zip "33100"}},
; :errors (#Error{:path [:tags 0], 
;                 :in [:tags 0], ;; <--- the set value paths are incorrect #294
;                 :schema keyword?,
;                 :value ":hotel",
;                 :loc {:row 2, :col 10, :end-row 2, :end-col 18}}
;           #Error{:path [:address :zip],
;                  :in [:address :zip],
;                  :schema int?,
;                  :value "33100",
;                  :loc {:row 5, :col 17, :end-row 5, :end-col 24}})
; :string "{:id \"Lillan\"
;           :tags #{:artesan :coffee \":hotel\"}
;           :address {:street \"Ahlmanintie 29\"
;                     :city \"Tampere\"
;                     :zip \"33100\"
;                     :lonlat [61.4858322, 23.7854658]}}
;          "}

;; string->edn, with malli string-coercion
(let [coerce (coercer Address (mt/string-transformer))]
  (coerce (slurp "schema.edn")))
; => nil
#2020-11-0914:02ikitommi@borkdude #294 is fixed in master and the edamame-walking works now and is bit simpler: 1. parse with edamame 2. prewalk twice to get both the original EDN and the path-vec -> loc lookup table 3. glue things together 4. kudos to @nilern for a working walker 5. https://gist.github.com/ikitommi/e3229a0bcef532d1fa032321713227d3#2020-11-0914:03ikitommi#2020-11-0914:05ikitommiit automatically binds a transformer named :edamame, so you can add custom decoding hints to schemas:
[:string {:decode/edamame str/upper-case}]
… and if sci is enabled, the schemas can be read from files too.
#2020-11-0918:53lmergeni have a particularly complex schema where the initialization of e.g. (m/validator) or (m/transformer) is fairly slow -- about 300ms. one way to deal with this would be to cache these -- is my understanding correct that using a registry will effectively do this ? or will i need to write my own caching layer on top of it ?#2020-11-0918:54lmergenor are registries just a very simple way of organizing stuff, without any pre-parsing going on ?#2020-11-0919:09ikitommi@lmergen the schema creation will get a 10x boost soon, the slow part being m/schema. If you add Schema instances into registry, it happends just once. Or you can just use a var:
(def Address (m/schema [:map [:street :string]]))
#2020-11-0919:10ikitommi… but, for super fast validation, you should just create the m/validator once and reuse that. it returns a pure and optimized function.#2020-11-0919:12lmergenright, I think I’ll just go for that last option. fairly often various of these validators are used in hot code paths, so I’ll probably write something to cache validators instead. #2020-11-0919:13lmergenbut if I wrap things in an m/schema call, it’ll already do a lot of preprocessing, right?#2020-11-0919:15ikitommiright. I’ll run some flamegraphs. just a sec.#2020-11-0919:24ikitommi
(time
  (prof/profile
    (dotimes [_ 50000]
      (m/validate [:map [:street :string]] {:street "hämeenkatu"}))))
;; "Elapsed time: 10472.153783 msecs"

(let [schema (m/schema [:map [:street :string]])]
  (time
    (prof/profile
      (dotimes [_ 500000]
        (m/validate schema {:street "hämeenkatu"})))))
;; "Elapsed time: 231.093848 msecs"

(let [validate (m/validator [:map [:street :string]])]
  (time
    (prof/profile
      (dotimes [_ 500000]
        (validate {:street "hämeenkatu"})))))
;; "Elapsed time: 59.743646 msecs"
#2020-11-1318:02hoynkMay I ask what profile lib you are using?#2020-11-1318:21ikitommimostly https://github.com/clojure-goes-fast/clj-async-profiler g https://github.com/hugoduncan/criterium#2020-11-1318:50hoynkthx#2020-11-0919:25lmergenright, this makes a lot of sense. #2020-11-0919:25ikitommi#2020-11-0919:26ikitommim/schema uses satisfies? which has a perf issue, most of the time spent there.#2020-11-0919:30ikitommior was it two orders of magnitude? satisfies? seems to take at least 95% of the time.#2020-11-0919:30ikitommi#2020-11-0919:37lmergenok, this is very helpful#2020-11-0919:41lmergenI really do find the validators to be significantly faster than spec validate — it’s about 3x faster for my fairly insane schema (the same that takes 300ms to parse). better yet, and this was unexpected: the generators are also much faster. I’m not 100% certain yet because whether this is because maybe Malli takes some shortcuts, but i seem to be able to avoid a few annoying gen/such-that? generators with Malli that causes a very large speed up. #2020-11-1009:21ikitommithis is interesting. tried to avoid such-that?, e.g. setting min & max when known, but have not tested against spec gen.#2020-11-0921:43ikitomminew flames with cache#2020-11-0921:45ikitommi
10472ms => 568ms (18x faster)
#2020-11-1007:18lmergen@ikitommi for what it's worth, i still got a huge performance increase by actually caching the validators as well.
(crit/with-progress-reporting
  (crit/quick-bench (m/validate schema value)))
;; => Execution time mean : 297.880813 ms

(def schema' (m/schema schema))
(crit/with-progress-reporting
  (crit/quick-bench (m/validate schema' value)))
;; => Execution time mean : 533.885193 µs

(def validator (m/validator schema))
(crit/with-progress-reporting
  (crit/quick-bench (validator value)))
;; => Execution time mean : 1.830348 µs
so it looks like about a 500x improvement by caching schemas, and then another 300x improvement by caching the validators
#2020-11-1007:21lmergeni suspect in your specific benchmark, the schema is fairly simple so then a larger share of the benchmark is actually about performing the validation#2020-11-1009:18ikitommi@lmergen there was a cljs-issue, just merged the cached satisfies. could you retry with the latest master?#2020-11-1009:19lmergensure!#2020-11-1009:19lmergen1 minute#2020-11-1009:20ikitommithere is still a lot of room for improvement for maps (`-parse-entries` is really slow) and for handling property-based registries. I would guess can make schema creation 2-5 times faster. But then again, after malli is used to validate schema properties & children, it will slow things down again.#2020-11-1009:29lmergen
(crit/with-progress-reporting
  (crit/quick-bench (m/validate schema value)))
;; before: => Execution time mean : 297.880813 ms
;; after:  => Execution time mean : 12.194964 ms

(def schema' (m/schema schema))
(crit/with-progress-reporting
  (crit/quick-bench (m/validate schema' value)))
;; before: => Execution time mean : 533.885193 µs
;; after:  => Execution time mean : 517.890217 µs


(def validator (m/validator schema))
(crit/with-progress-reporting
  (crit/quick-bench (validator value)))
;; before: => Execution time mean : 1.830348 µs
;; after:  => Execution time mean : 1.952607 µs
so while m/validate got ~ 20x faster, caching the actual validator is still much, much faster
#2020-11-1009:40lmergeni'm caching the explainers in my own defn macro, but it requires quite a bit of macro magic to make this work, so i was looking for a more generic way to make this happen -- possibly some kind of registry#2020-11-1009:43ikitommiwhat should be in the registry? validator + explainer + generator + decoder(s) + encoder(s) + …?#2020-11-1009:44lmergenif possible, i'd say all of them yes#2020-11-1009:44ikitommiin reitit, there is a Coercion protocol to cache things relevant there: https://github.com/metosin/reitit/blob/master/modules/reitit-malli/src/reitit/coercion/malli.cljc#2020-11-1009:45lmergenright, so then you lazily cache things#2020-11-1009:45lmergenwhich would be the best middle-ground#2020-11-1009:47ikitommione would be to add a wrapper Schema impl, that is returned from registry instead of the real one. And that impl would have a cache -> first call to -validate would store the validator.#2020-11-1009:47lmergenoh i see#2020-11-1009:47ikitommicould be just an option to the registry to return caching proxys instead of normal ones…#2020-11-1009:47lmergenyes#2020-11-1009:47lmergenthis would be very effective#2020-11-1009:48lmergeni'll experiment with this approach#2020-11-1009:50ikitommi… actually, just a new option key that m/schema uses would do fine (to wrap the returned thing if the option is present)#2020-11-1010:02lmergenso then you cache it inside the actual schema, rather than a wrapper around it ?#2020-11-1010:06ikitommiI would wrap it outside, e.g. the return value wrapped#2020-11-1010:06lmergenah, right -- and the option to m/schema would then tell it whether to return the wrapped schema or the "regular" schema#2020-11-1010:09ikitommiyes. Or there could be a memoized-schema etc. as a separate fn? (-> :string m/schema m/memoized)#2020-11-1010:09lmergenyes#2020-11-1010:09lmergenwell that's a detail#2020-11-1010:09lmergenlet me experiment with creating that memoized / cached schema in the first place#2020-11-1016:52lmergen
(crit/with-progress-reporting
  (crit/quick-bench (m/validate schema value)))
;; => Execution time mean : 11.782073 ms

(def schema' (memoized-schema (m/schema schema)))
(crit/with-progress-reporting
  (crit/quick-bench (m/validate schema' value)))
;; => Execution time mean : 2.095245 µs
@ikitommi conceptually it seems to be working like a charm
#2020-11-1016:55lmergenexactly which schema am i supposed to wrap here -- it's just the regular malli.core/Schema, right ? the into-schema is meant more for building a hierarchy of parent/child schemas ?#2020-11-1016:57ikitommiyes Schema. IntoSchema is the factory-protocol for creating a Schema out of the Schema AST, each Schema is responsible for it’s own props & children.#2020-11-1016:57lmergenright#2020-11-1016:57lmergenawesome#2020-11-1016:57lmergeni'll send a PR once i have all the functions working#2020-11-1115:54Maciej FalskiHey, I think I found a bug while trying to get malli coercion working with reitit. Basically it comes to this:
; based on  example
(def Over6
  (m/-simple-schema
    {:type :user/over6
     :pred #(and (int? %) (> % 6))
     :type-properties {:error/message "should be over 6"
                       :decode/string mt/-string->long
                       :json-schema/type "integer"
                       :json-schema/format "int64"
                       :json-schema/minimum 6
                       :gen/gen (gen/large-integer* {:min 7})}}))

(mu/closed-schema Over6)

=> #'user/Over6
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :user/over6}
I can raise an issue in GH if you can confirm it.
#2020-11-1214:24Rodrigo A. RoveriFolks, I have a begginer's question. I would like to test Malli, so I created a new lein project and included malli in the dependencies. I was able to run lein repl, but was unable to follow the examples on the doc page. (see image below) Am I missing something? Some pre-requisite? I tested on these two versions "0.0.1-SNAPSHOT", "0.2.1".#2020-11-1214:24Rodrigo A. Roveri#2020-11-1214:34borkdudeWhat does the exception preceding your REPL expressions say?#2020-11-1214:53Rodrigo A. Roveri#2020-11-1214:53Rodrigo A. Roveri@borkdude#2020-11-1214:54borkdudeHow are you starting this repl? what is in your project.clj?#2020-11-1214:56Rodrigo A. RoveriOh oh. I think maybe I know the answer. My project is also named "malli":man-facepalming:#2020-11-1214:57Rodrigo A. Roveri#inception#2020-11-1214:58Rodrigo A. RoveriThank you very much. Keep up the good work. I will run some tests. The project seems really interesting!#2020-11-1217:13ikitommi@maciej.falski oh, that’s bad, didn’t realize that unregistered Schemas can’t be walked with the current impl of m/schema-walker, as it tries to recreate the schmas using m/type, which in this case is :user/over6 and not found in the registry.#2020-11-1217:13ikitommiMight need to add a -copy method into Schema protocol.#2020-11-1217:14ikitommigood thing is, that a per-schema -copy is most likely near to no-op, which makes Schema walking easily order(s) of magnitude faster.#2020-11-1217:37Maciej FalskiNot sure if it’s about the registry, ie:
(mr/set-default-registry!
  {:string (m/-string-schema)
   :over6 Over6})
=> #object[malli.registry$simple_registry$reify__4072 0x1f74f3fd "
#2020-11-1217:42ikitommi@maciej.falski you should register it with :user/over6 so it matches the Schema definition#2020-11-1417:26ikitommiSchema knows now the IntoSchema that creates it. Fixes the issue @maciej.falski and makes schema copying (= all utils) faster.#2020-11-1419:47ikitommitwo ways to create reusable rules for mutually exclusive keys for maps: 1. just data:
(defn exclusive-keys [keys]
  [:fn {:error/message (str "the following keys are mutually exclusive: " (str/join ", " keys))}
   (fn [m] (not (every? (partial contains? m) keys)))])

(-> [:and
     [:map
      [:foo {:optional true} :int]
      [:blabla {:optional true} :int]
      [:hah {:optional true} :int]
      [:bar {:optional true} :int]]
     (exclusive-keys [:hah :bar])
     (exclusive-keys [:hah :foo])]
    (m/explain {:foo 1 :hah 2})
    (me/humanize))
; => #:malli{:error ["the following keys are mutually exclusive: :hah, :foo"]}
2. new Schema impl (with pretty form):
(def Exclusive
  (m/-simple-schema
    (fn [_ [keys]]
      {:type 'Exclusive
       :min 1
       :max 1
       :pred (fn [m] (not (every? (partial contains? m) keys)))
       :type-properties {:error/message (str "the following keys are mutually exclusive: " (str/join ", " keys))}})))

(-> [:and
     [:map
      [:foo {:optional true} :int]
      [:blabla {:optional true} :int]
      [:hah {:optional true} :int]
      [:bar {:optional true} :int]]
     [Exclusive #{:hah :bar}]
     [Exclusive #{:hah :foo}]]
    (m/explain {:foo 1 :hah 2})
    (me/humanize))
; => #:malli{:error ["the following keys are mutually exclusive: :hah, :foo"]}
#2020-11-1419:49ikitommino macros = awesome#2020-11-1419:52ikitommithe latter is the not-well-documented reagent-style thing, any IntoSchema can be used in first position in Schema AST and -simple-schema allows one to read the properties & children at creation time and return a Schema instance based on those.#2020-11-1419:52ikitommiit’s bit like derived types schemas?#2020-11-1509:30ikitommiwrote an issue for discussing about auto-updating with container schemas: https://github.com/metosin/malli/issues/304. Comments most welcome.#2020-11-1511:37ikitommifunction schemas, comments welcome: https://github.com/metosin/malli/issues/125#issuecomment-727555388#2020-11-1511:40borkdudeI think you are ignoring the fact that different arities can have different return types/schemas#2020-11-1511:41borkdudeI think clojure.spec is also making this too difficult. In clj-kondo I chose to define the spec per arity#2020-11-1511:41borkdudeIt is more verbose, but at least easy to verify#2020-11-1511:42ikitommigood point, the schema also forces the return to be the same for all arities.#2020-11-1511:42borkdudereally?#2020-11-1511:43ikitommiI guess there are lot of examples in the core where the different arities return different things?#2020-11-1511:43ikitommiyes.#2020-11-1511:44borkdudeconfirmed: > - The output schema always goes on the fn name, not the arg vector. This means that all arities must share the same output schema.#2020-11-1511:45borkdudewell, clojure.core/map, filter, etc, is an example that have different return types for different arities. I kinda wish that the transducer arity was just a different version like clojure.core/mapping but that ship has sailed. I often make mistakes with this#2020-11-1511:45borkdudelook at the spec that will result from this: https://github.com/borkdude/speculative/blob/master/src/speculative/core.cljc#L297-L302#2020-11-1511:46borkdudeI think this is not ergonomic at all. spec has to do backtracking etc, to match the right arity.#2020-11-1511:46borkdudeimo the fn spec should match the structure of the defn args+bodies#2020-11-1511:49ikitommigood point. what would be a good malli definition for this:
(defn fun
  ([x] x)
  ([x y] [x (* x x)]))
#2020-11-1511:50ikitommiclj-kondo has a good syntax for the different arities, use that’ish?#2020-11-1511:50ikitommisomething like:
(m/=> fun {:arities {1 {:output int?
                        :input [:tuple int?]}
                     2 {:output [:tuple int? pos-int?]
                        :input [:tuple int? int?]}}})
#2020-11-1511:51borkdudeThat's how I do it in clj-kondo yes.#2020-11-1511:52borkdudeNot sure if that's the best, but I optimized for matching speed, so clj-kondo can find the right arg types fast#2020-11-1511:53borkdudeas for defn syntax, maybe:
(defn fun
  ([x :- int?] :- int?
   x)
  ([x :- int? y :- string?] :- [:tuple string? int?]
   [y (* x x)]))
#2020-11-1511:53borkdudeso the return type directly after the arg vec#2020-11-1511:54ikitommi👍#2020-11-1511:57ikitommithe extracted schemas could be in the compact/short :=> format, which is always 1-arity:
(defn fun1 [x] (* x x))

;; short
(m/=> fun1 [:=> int? [:tuple pos-int?]])
(defn fun
  ([x] (fun x x))
  ([x y] [x (* x x)]))

;; short
(m/=> fun [:or
           [:=> int? [:tuple int?]]
           [:=> [:tuple int? pos-int?] [:tuple int?]]])
#2020-11-1511:59ikitommithe long versions:
(defn fun1 [x] (* x x))

;; long
(m/=> fun1 {:arities {1 {:input int?
                         :output [:tuple pos-int?]}}})
(defn fun
  ([x] (fun x x))
  ([x y] [x (* x x)]))

;; long
(m/=> fun {:arities {1 {:output int?
                        :input [:tuple int?]}
                     2 {:output [:tuple int? pos-int?]
                        :input [:tuple int? int?]}}})
#2020-11-1511:59borkdudecould work yes#2020-11-1511:59borkdudedoesn't fn spec hinge on sequence specs which are not exposed yet?#2020-11-1512:01ikitommiyes, those are needed for varags. we just had an internal tech-talk on friday, did a plan how to get the sequence schemas & schema parsing out. takes few days to make that good.#2020-11-1512:01ikitommifirst demo of the function schemas will be with non-vargargs. enough to get feedback & start with the clj-kondo integration.#2020-11-1516:09ikitommi
(require '[malli.schema :as ms])

(ms/defn ^:always-validate fun :- [:tuple int? pos-int?]
  "returns a tuple of a number and it's value squared"
  ([x :- int?] :- any? ;; arity-level override
   (fun x x))
  ([x :- int?, y :- int?] ;; uses the default return
   [x (* x x)]))

(clojure.repl/doc fun)
; -------------------------
; demo/fun
; ([x] [x y])
;   
;   [:-> [:tuple int?] any?]
;   [:-> [:tuple int? int?] [:tuple int? pos-int?]]
; 
;   returns a tuple of a number and it's value squared
#2020-11-1516:16ikitommifull meta:
(ms/defn square :- pos-int?
  [x :- int?]
  (* x x))

(meta #'square)
;{:schema [:or [:-> [:tuple int?] pos-int?]],
; :ns #object[clojure.lang.Namespace 0x3c5f3ba8 "demo"],
; :name square,
; :file "/Users/tommi/projects/metosin/malli/src/malli/schema.cljc",
; :column 1,
; :raw-arglists ([x :- int?]),
; :line 64,
; :arglists ([x]),
; :doc "\n[:or\n [:-> [:tuple int?] pos-int?]]"}
#2020-11-1520:28ikitommi#2020-11-1606:04ikitommifirst draft of m/=>, to annotate existing functions. now - just with the compact/mallli notation, could support the map-version too. Kinda like it as it’s as long as defn, to the two can be aligned.#2020-11-1606:05ikitommi… supports multi-artities with different return for each.#2020-11-1610:50ordnungswidrigmagic#2020-11-1708:45ikitommithere are some gaps in the malli->clj-kondo integration, any insights welcome: https://github.com/metosin/malli/blob/cde74871ee5edc1525344f7a6e62d54fb1b00f5b/src/malli/clj_kondo.cljc. marked with ;;??#2020-11-1708:45ikitommicurrently:
(require '[malli.clj-kondo :as mc])

(mc/transform
  [:map
   [:id string?]
   [:tags {:optional true} [:set keyword?]]
   [:address
    [:map
     [:street string?]
     [:city string?]
     [:zip {:optional true} int?]
     [:lonlat [:tuple double? double?]]]]])
;{:op :keys
; :opt {:tags :set}
; :req {:id :string
;       :address {:op :keys
;                 :opt {:zip :int}
;                 :req {:street :string
;                       :city :string
;                       :lonlat [:double :double]}}}}
#2020-11-1708:51ikitommibtw, would be great if clj-kondo supported :min and :max for numbers & collections. nice demo about “more than types”, something like:
(ms/defn times :- :int
  "times"
  [x :- [:int {:min 3}]]
  (* x x))

(times 2)
;; ^:--- clj-kondo error of "number should be at least 3"
#2020-11-1709:25borkdudeDon't know if this a common enough problem in function calls. I haven't encountered this much I think. If there's a domain specific need to check this, one can also write a hook#2020-11-1709:32ikitommiok. just wondering as clj-kondo already supports map-op-syntax, so if :int could be defined as {:op :int}, it would allow adding easily new keys that would be easy to check, e.g. malli could emit {:opt :int, :min 3} from the example. Not sure how useful that would be, but: • number min & max might be easy to implement on clj-kondo side? • would be a great demo, as you can’t easily present these with simple typed langs liike Java#2020-11-1709:32ikitommibut, I don’t need this for real, just thinking aloud about a kick-ass demo 🙂#2020-11-1709:33borkdudeit has :pos-int and :nat-int which map to the clojure predicates. it basically only has that which exists as predicates in clojure 1.9+#2020-11-1709:36borkdude@ikitommi Feel free to try out things with the clj-kondo type system. You can build a custom clj-kondo for your demo possibly#2020-11-1709:36borkdudeit's all in clj-kondo.impl.types#2020-11-1709:36ikitommithanks, might give it a shot.#2020-11-1709:42borkdude@ikitommi Ah, I can see why your approach might be problematic: https://github.com/borkdude/clj-kondo/blob/master/src/clj_kondo/impl/types.clj#L157-L163 So the literal value is mapped to a type tag. And then during checking that is resolved very cheaply, not executing any predicates, but just checking a keyword identity or a super simple graph, e.g. a pos-int is also an int#2020-11-1709:44borkdudeso the literal value is erased as it were#2020-11-1718:47HankstenbergHi guys, just getting my feet wet with malli. Quick question: when I try to transform a map to JSON-Schema, all the map's keywords are still Clojure keywords and thus invalid syntax in JSON-Schema. Sure, I could post-process it, but is this on purpose? What's the best way to get proper JSON-Schema?#2020-11-1719:03ikitommi@marcus.poparcus I think it would be better to emit strings directly. PR Welcome.#2020-11-1820:34HankstenbergThe simplest way to do it would probably be to wrap the output in clojure.data.json/write-str, looks pretty ugly of course and becomes useless in clojureland. I guess it really depends on what consumes the output afterwards. Maybe that should be handled case by case. How about an additional function "transform-json-str"? Comes with the new dependency of clojure.data.json, though. Looking at the source code of clojure.data.json it's not trivial to generate valid JSON manually.#2020-11-2001:36fmnHi guys, is there any way to get all of the values defined in :enum ? I'm trying to do some testing and it would be great if I could just iterate through it rather than hard-coded it. I guess I could just (next (malli.core/form my-enums)) . Just trying to find a more consistent way since this breaks if I add options to it.#2020-11-2004:46ikitommi@funyako.funyao156 try (m/children MyEnun)#2020-11-2215:58ikitommihi all. comments on the function schema syntax? some alternatives:
;;
;; many ways to present function schemas
;; - 2 ints to int
;; - int to int
;; - no args to int
;; - no args to irrelevant
;;

;; 1: current
[:=> [:tuple int? int?] int?]
[:=> [:tuple int?] int?]
[:=> [:tuple] int?]
[:=> [:tuple]]

;; 2: shortcut
[:=> [int? int?] int?]
[:=> [int?] int?]
[:=> [] int?]
[:=> []]

;; 3: separator
[:fn int? int? :=> int?]
[:fn int? :=> int?]
[:fn :=> int?]
[:fn :=>]
#2020-11-2303:58rutledgepaulvI personally do not mind the more verbose version. I am wary of adding too many syntactic shortcuts since sometimes they begin to collide in unfortunate ways that make it more difficult to write correct tooling. But maybe you already have easy ways to quickly convert them into a normal form?#2020-11-2306:51ikitommino tools to convert, totally agree that there should not be extra syntaxes in malli core.#2020-11-2307:08Martín VarelaI like the 3rd version best, seems easier to read than the other ones.#2020-11-2307:17ikitommi@U95NTJT4H that is basically Ghostwheel (spec) / Aave (malli) do:
(>defn bad-return-val
  [x y]
  [int? int? => string?]
  (+ x y))
#2020-11-2307:19ikitommione more:
;; 4: via properties
[:=> {:input [:tuple int? int?]
      :outut int?}]
[:=> {:input [:tuple int?]
      :output int?}]
[:=> {:output int?}]
:=>
#2020-11-2307:20Martín VarelaI haven't used those, but it does seem more readable than the alternatives (I guess this is a very subjective thing, anyway). I guess I'm also with @U5RCSJ6BB in not minding a bit more verbosity#2020-11-2307:20Martín Varelaterse is really cool, until it isn't... 🙂#2020-11-2307:23ikitommidata-specs for spec-tools for a great idea for really simple things. But as soon as one needed something non-trivial, it became a burden. Same with spec2, s/select syntax is awesome, until you need something inlined specs in it.#2020-11-2307:23Martín VarelaBTW, is the idea to provide the full power of malli for these? If I think of my use cases, the most likely ones would probably be rather complicated crap data (nested, with constraints, etc..), so it'd be cool to be able to use a registry or similar#2020-11-2307:24Martín Varelathey may become rather unreadable very fast, if inlined as such#2020-11-2307:25ikitommi:=> is just a normal Schema, so generators, validators etc. work normally#2020-11-2307:25Martín Varelaok#2020-11-2307:31Martín VarelaNot entirely unrelated to this, the other day, while trying to figure out that reitit issue, I thought the problem was that reitit wasn't picking up the registry, so I ended up writing a bit of meander term rewriting code that allows you to define schemas based on other schemas (where you'd normally use a registry), and compile those to malli "primitives". So you get both the concision and explicitness, in a way.#2020-11-2311:47rutledgepaulvSuppose another question is should a "irrelevant" return be a special case or should you just use a 'any? schema#2020-11-2216:03ikitommiI think as Aave already provides the short variant, so the official (malli-like) syntax doesn’t have to be that terse, just formal. But the 1 is still.. ugly.#2020-11-2306:53ikitommiplaying with :=> generators. also good peek at the syntax options (using children vs properties):
(def Age
  [:int {:min 18, :max 100}])

(def User
  [:map
   [:name string?]
   [:age Age]
   [:skills [:set [:enum "clj" "cljs" "rust" "go" "cobol"]]]])

;; option1: childs
(def create-user
  [:=> {:gen/=> '(fn [[age] user] (assoc user :age age))}
   [:tuple Age] User])

;; option2: properties
(def create-user
  [:=> {:gen/=> '(fn [[age] user] (assoc user :age age))
        :input [:tuple Age]
        :output User}])

(mg/generate User {:size 10})
; => {:name "Szf5sN", :age 21, :skills #{"cobol" "rust" "cljs" "clj" "go"}}

(def create-user-gen (mg/generate create-user {:size 10}))

(create-user-gen 46)
; => {:name "Itu96", :age 46, :skills #{"cobol" "rust" "cljs" "clj" "go"}}
#2020-11-2321:31ikitommialso, validating functions against mallidefinitions:
(def Age
  [:int {:min 18, :max 100}])

(def User
  [:map
   [:name string?]
   [:age Age]
   [:skills [:set [:enum "clj" "cljs" "rust" "go" "cobol"]]]])

(mg/generate User {:size 10})
; => {:name "Szf5sN", :age 21, :skills #{"cobol" "rust" "cljs" "clj" "go"}}

(def create-user [:=> [:tuple Age] User])

(m/validate
  create-user
  (fn [age] {:name "elephant", :age age, :skills #{"cobol"}})
  {::m/=>validator mg/=>validator})
; => true

;; invalid return
(m/validate
  create-user
  (fn [age] {:name "elephant", :age age, :skills #{"PERL"}})
  {::m/=>validator mg/=>validator})
; => false

;; invalid arity
(m/validate
  create-user
  (fn [age _extra] {:name "elephant", :age age, :skills #{"cobol"}})
  {::m/=>validator mg/=>validator})
; => false
#2020-11-2415:18heliosIt's not clear how i could say that i want a non-empty map in my schema, that also works with generators 🙂#2020-11-2415:27helios
(mg/generate [:and
              [:fn not-empty]
              [:map-of
               [:string {:gen/gen gen/string-ascii}]
               [:string {:gen/gen gen/string-ascii}]]])
this ofc works for validation but not with generators
#2020-11-2415:53ikitommi@U0AD3JSHL try reversing that, :map-of as first child. :and generates based on first, and narrows with the rest.#2020-11-2415:54heliosuh, nice one 🙂#2020-11-2421:19Hankstenbergmalli is absolutely awesome, thanks a lot guys! 👍#2020-11-2421:29ordnungswidrigIs there a “canonical representation” for malli? I guess it’s the “vector” syntax as opposed to the “map” syntax, right?#2020-11-2505:45steveb8n@ikitommi is ::keys in this line a bug? https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L337#2020-11-2505:48ikitommi@U0510KXTU not a bug, it uses qualified keys for configuration: https://github.com/metosin/malli/blob/master/test/malli/transform_test.cljc#L207#2020-11-2505:49steveb8nok. TIL a new destructing syntax 🙂#2020-11-2505:50ikitommiyes, a handy way to do that. Also, {:keys [::json-vectors]} would work.#2020-11-2505:53ikitommi@ordnungswidrig right, it the vector syntax for now.#2020-11-2511:37Elsois there something like a best-effort implementation of a json-schema to malli converter out there?#2020-11-2515:19ikitommi@d.eltzner012 don't think so, there is an old PR, but lot's of todos and last update from jul.#2020-11-2515:19ikitommihttps://github.com/metosin/malli/pull/211#2020-11-2515:22rutledgepaulvI have a half-baked implementation of this for converting kubernetes swagger into malli. I'm using malli to write a clojure kubernetes client with client side validation of operations (similar to aws-api). It handles recursive schemas at least but some of the kubernetes swagger is on the janky side so i'm sure it's not properly general. The whole library is still early stages. https://github.com/RutledgePaulV/kube-api/blob/master/kube-api-core/src/kube_api/core/swagger/malli.clj#2020-11-2618:33ikitommi👍#2020-11-3008:24Elso👍 very cool still, I'll have a look at it#2020-11-2611:02HankstenbergThe more I look at it the more I want to use malli schemas as a single source of truth for everything. Are there any resources about generating ui models from malli schemas? I think the idea to integrate it with Domino was floating around once, but the current version doesn't mention malli. How else could it be done? My go-to framework for UI stuff is usually re-frame. Malli could be integrated so that re-frame's db is generated from it. And to validate state changes, of course. It could also be bound to visual representations of actual components I guess. Would be grateful for any reference to existing work in this area.#2020-11-2614:32rutledgepaulvI have played with this a bit. I eventually came to a realization that I should first transform malli into an intermediate representation prior to generating the UI state because there are additional things you'll want to know about each node, like whether the field is "dirty" and has been "visited", etc. So in other words.. first I would go through the work of figuring out a sensible IR and generating forms from that, and then work backwards and generate that initial IR from malli.#2020-11-2614:35rutledgepaulvThat said, my initial attempt of using malli directly (and still struggling with how to do the reagent cursors) is here: https://rutledgepaulv.github.io/ui-kit/#!/ui_kit.forms https://github.com/RutledgePaulV/ui-kit/blob/master/src/cljs/ui_kit/visitors.cljs#2020-11-2618:53ikitommiHave seen multiple prototypes. Did also a demo into on of our projects. There is malli.util/subschemas helper which takes a Schema and returns an ordered list of all the schema paths and Schemas in those, handling also collection schemas. One place to start. Also, one can post-walk the Schemas and wrap all schmas into something that can be used to strore the form state (dirty, pristine, etc). hope this helps.#2020-11-2618:59ikitommi#2020-11-3006:53HankstenbergThanks a lot for the input! I'm still trying to wrap my head around everything. I think Datascript (re-posh)+datasync are the right tools to connect front-end state and back-end state, I think I will try to find a way to make this stack (malli-)schema-driven.#2020-11-2618:53ikitommiHave seen multiple prototypes. Did also a demo into on of our projects. There is malli.util/subschemas helper which takes a Schema and returns an ordered list of all the schema paths and Schemas in those, handling also collection schemas. One place to start. Also, one can post-walk the Schemas and wrap all schmas into something that can be used to strore the form state (dirty, pristine, etc). hope this helps.#2020-11-2619:03rutledgepaulvWonder if there's an opportunity to improve error messages here:#2020-11-2619:03rutledgepaulv
(def stream
  (logs client
        {:operation "readCoreV1NamespacedPodLog"}
        {}))
Execution error (ExceptionInfo) at kube-api.utils/validation-error (utils.clj:87).
Invalid request.
{:path-params ["missing required key"]}

(def stream
  (logs client
        {:operation "readCoreV1NamespacedPodLog"}
        {:path-params {}}))
Execution error (ExceptionInfo) at kube-api.utils/validation-error (utils.clj:87).
Invalid request.
{:path-params
 {:name ["missing required key"], :namespace ["missing required key"]}}
#2020-11-2619:04rutledgepaulvif there's a missing key, insert the key with an empty value and run validation on that empty value too so you can report not only the missing collection but also what needs to be in the collection?#2020-11-2619:04rutledgepaulvprobably breaks down in some edge cases.. thinking#2020-11-2619:06ikitommiyou could run default-value-transformer to add empty maps first.#2020-11-2619:08ikitommi
(m/decode
  [:map
   [:path-params [:map
                  [:name :string]
                  [:description :string]]]]
  nil
  (mt/default-value-transformer
    {:defaults {:map (constantly {})
                :string (constantly "")}}))
; => {:path-params {:name "", :description ""}}
#2020-11-2619:14rutledgepaulvthanks! yes, that's working for me#2020-11-2914:05joshkhnoob question, but how can i validate/generate non-blank strings? with spec i would use (complement clojure.string/blank?) .
(m/validate [:map [:name [:and string? (complement clojure.string/blank?)]]] {:name ""})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema #object[clojure.core$complement$fn__5669 0x394ebe0b "
#2020-11-2914:39joshkhthis validates, but fails on generation
(m/validate [:map [:name [:fn (complement clojure.string/blank?)]]] {:name "test"})
#2020-11-2915:13eval2020@U0GC1C09L the README has [:string {:min 1}] as example.#2020-11-2915:18joshkhnot sure how i missed that. thanks eval2020.#2020-11-2915:35ikitommiI think there should be a built-in complement.#2020-11-2915:35ikitommidoesn’t exist, but should.#2022-07-0208:52Eric Dvorsak[:string {:min 1}] isn't the same, the string could still be blank, eg " "#2020-11-2915:36ikitommimerged into master the current status of function schemas, alpha, feedback welcome! the guide is here: https://github.com/metosin/malli#function-schemas#2020-11-2915:36ikitommiand the PR https://github.com/metosin/malli/pull/306#2020-11-2915:37ikitommialso has now the full clj-kondo integration !!#2020-11-2915:38ikitommiwill make a separate PR of the optional plumatic-style helpers.#2020-11-2915:40ikitommithere are now var instrumentation helpers atm, only shipped use case is the clj-kondo type checker integration. Any ideas what would be a good api to instrument functions in dev to run the input & output checks?#2020-11-2915:40ikitommiany ideas about clojure.test helpers for malli?#2020-11-2915:41ikitommidev-time tooling has not been the goal for malli (as it’s the primary goal for spec already), but happy to add things to that direction too.#2020-11-2915:42ikitommiwith spec, have used https://github.com/jeaye/orchestra, I guess like everyone else 🙂#2020-11-3007:22HankstenbergQuick question: Is is possible to combine malli and spec-tools in order to get both, the JSON schema and the API schema? Do you plan to extend spec-tools or malli? Actually it has to be malli right? Yesterday I realized how awesome programatically updating malli specs is. You can have a complex user schema with an ":active" boolean in in and all you need to do to specify an inactive user is to say (def inactive-user (mu/assoc user :active [:enum false]). It's like a language-agnostic type that would be perfectly exportable to e.g. openapi3 and JSON form to make use of the tools available for these standards. (trying to find good reasons to introduce clojure with malli as a way to generate openapi/json schema in our company that uses them).#2020-12-0518:14ikitommiSlow answer 😉 : I’m not sure I understand the question, but you can generate JSON Schema from malli. Glad you like it.#2020-12-0707:07HankstenbergYea, I know and I'm loving it. I'd like to experiment a bit with API-Driven development at our company. OpenApi is nice, but who wants to write JSON/YAML by hand when you can write edn instead and programmatically create JSON Schema and OpenAPI-Specs from that. The JSON Schema part is covered nicely by malli, which is wonderful, because we can utilize the nice ecosystem that uses JSON Schema already AND have the power of property-based testing. But what about speccing the endpoints? There is spec-tools, but spec-tools uses clojure.spec instead of malli. I imagine that I could e.g. specify an "Order" schema and a "Processed Order" schema in order to specify an entity and an entity in a certain state (with "processed" set to "true" and a "processed-on" field set to some date and so on. With the ability to even spec functions with malli, we have all the building blocks to extend malli's capabilities to also specify the actual API: (def =>process-oder [:=> [:tuple entities/order events/process-order] entities/processed-order]) Now all that's missing is to map this to e.g. a POST request and generate a complete openapi-spec from it (schema+end points), so that e.g. a REST-Client can be generated from this code. At least that's how I would imagine it. Maybe I'm on the wrong track here, though.#2020-12-0707:36ikitommihave you looked at reitit-malli? here’s an example: https://github.com/metosin/reitit/tree/master/examples/ring-malli-swagger#2020-12-0708:59HankstenbergThat's the best possible reply. Yes, there already is something and btw it's even better than what you had in mind. Thanks a bunch! 👍#2020-12-0115:05mynomotoAbout the clj-kondo integration, is malli annotations compatible with the clj-kondo ones? As in if something is typed string on malli and is passed to a core function that only accepts numbers, will it show as a warning?#2020-12-0115:06borkdude@mynomoto this doesn't work automatically, you should execute some code to make malli spit out type information to a clj-kondo config directory and you should opt in to this config dir via :config-paths#2020-12-0115:09mynomotoI'm assuming that the developer did the correct project configuration. The question is more about if a malli string is a clj-kondo string when doing the type inference.#2020-12-0115:10borkdude@mynomoto it should: https://github.com/metosin/malli/blob/23772defaa6223529edc316fd1ba8fb0c87ae3f0/src/malli/clj_kondo.cljc#L23#2020-12-0115:13mynomotoThis is great, thanks! I was wondering if a project to add malli type annotations to core functions would be useful for type inference but I'm glad that it is not necessary.#2020-12-0413:39Adrian SmithHey I'm trying out the clj-kondo integration, I've used the latest sha for malli in my deps.edn and have a working clj-kondo installation, I can see some clj-kondo configuration has been emitted by malli but the plus example doesn't appear to be erroring, is this ready for looking at?#2020-12-0515:33rutledgepaulvis there a recommended way to write a function that will dispatch to an implementation according to which of several schemas match my input arguments? I can construct my own of course that just does a linear search, and i could create a multi schema to compose them together (though i would need to use a compound discriminator for my case). I guess what i'm wondering is if there's a way to do pattern matching using malli schemas, but preferably in an open way like multimethods so i can just accrete new cases instead of modifying a case expression.#2020-12-0515:35rutledgepaulvmaybe i'm just rubbing up against open arbitrary predicate dispatch at that point. hm#2020-12-0515:37rutledgepaulvperhaps i could add an optional attribute on each schema (in my case they are open maps so this is fine) and supply a :default attribute. then i can run a default transformer on my inbound value to inject a "type" and from there just use regular multimethods to dispatch on that attribute.#2020-12-0515:57rutledgepaulvyeah that works. sorta rough and not sure the macro is a good idea but in case anyone is curious:#2020-12-0515:57rutledgepaulv
(defn dispatchable [dispatch-key schema]
  (mu/update-properties
    schema
    (fn [props]
      (let [dispatch-decoder (fn [value] (with-meta value {:dispatch dispatch-key}))]
        (assoc-in props [:decode/dispatch] {:leave dispatch-decoder})))))

(defn dispatch-key [schema value]
  (some-> (m/decode schema value (mt/transformer {:name :dispatch})) meta :dispatch))

(defmacro defdispatchable [symbol & body]
  `(def ~symbol (dispatchable ~(name symbol) (do 
#2020-12-0515:57rutledgepaulv
(handle-auth {:tokenFile "st"})
token file!
=> nil
(handle-auth {:client-key-data "st" 
              :client-certificate-data "sfd"})
client key!
#2020-12-0516:20rutledgepaulvActually, i think this is much better:#2020-12-0516:20rutledgepaulv
(def schema-resolver-transformer
  (mt/transformer
    {:default-decoder
     {:compile (fn [schema _]
                 (fn [value]
                   (if (instance? IObj value)
                     (vary-meta value assoc :resolved schema)
                     value)))}}))

(defn resolve-schema [schema value]
  (some-> (m/decode schema value schema-resolver-transformer) (meta) :resolved))

(def token-file-auth
  [:map {:dispatch :token-file}
   [:tokenFile :string]])

(def client-key-auth
  [:map {:dispatch :client-key}
   [:client-key-data :string]
   [:client-certificate-data :string]])

(def combined
  [:or token-file-auth client-key-auth])

(defn dispatch-fn [context]
  (let [schema (resolve-schema combined context)]
    (:dispatch (m/properties schema))))

(defmulti handle-auth #'dispatch-fn)

(defmethod handle-auth :token-file [context]
  (println "token file!"))

(defmethod handle-auth :client-key [context]
  (println "client key!"))
#2020-12-0516:30rutledgepaulvTommi, let me know if there's a better way to do this ^. Providing a way to resolve the concrete schema that is used for each node in a tree of data allows me to put useful data into the schema properties and recover that data for a given value so that i can act on it (like dispatching to different functions in my code based on which schema matched). Here i am attaching the schema to the data as metadata (which makes sense but will not work for values that don't implement IObj)#2020-12-0518:08ikitommi@sfyire @borkdude I think one needs to add extra entry into :config-paths into .clj-kondo/config.edn. I have the following:
✗ cat .clj-kondo/config.edn
{:config-paths ["configs/malli"]}
forgot all about that, will add it to README if that is the required glue here.
#2020-12-0518:12ikitommihttps://github.com/metosin/malli/commit/da45eceaea3e5b34426027bf7836e7cbe841cc97#2020-12-0518:08ikitommiwould be great if that was not needed and it would just work.#2020-12-0518:09ikitommie.g. with that, clj-kondo pics up the emitted file, looking something like this:
✗ cat .clj-kondo/configs/malli/config.edn
{:lint-as #:malli.schema{defn schema.core/defn}
 :linters {:type-mismatch {:namespaces {user {square {:arities {1 {:args [:int], :ret :nat-int}}}
                                              plus {:arities {1 {:args [:int], :ret :int}
                                                              2 {:args [:int :int], :ret :int}}}}}}}}
#2020-12-0518:31ikitommi@rutledgepaulv your second options looks interesting, tagging values. I most likely would do it like that. One problem I see with this approach is that the meta-data can get out-of-sync: if your say (dissoc data :tokenFile) , it still is tagged as :token-file, but it not anymore valid against that schema. Might not be a problem if you just use it once, but relying of :resolved meta in general has not guarantees. The Schema parsing is currently WIP (first release coming soon), with that, you will have a variant to :or (and :alt), which will have named branches, like spec does, so you can say ~about this:
(def token-file-auth
  [:map
   [:tokenFile :string]])

(def client-key-auth
  [:map
   [:client-key-data :string]
   [:client-certificate-data :string]])

;; using named :or variant
(def combined
  [:or*
    [:token-file token-file-auth]
    [:client-key client-key-auth]])

(m/parse combined {:token-file "kikka"})
; => #Branch{:key :token-file 
             :value {:token-file "kikka"}}
#2020-12-0518:34rutledgepaulvthanks! the schema parsing stuff looks like that would address my issue too#2020-12-0518:34ikitommiyou could also use m/encode and return a tuple yourself, with both the schema/name and the value.#2020-12-0518:35rutledgepaulvyeah. in my case i only need to know the concrete version of the top level schema so tuples would be fine#2020-12-1008:23Kari Marttilahttps://www.youtube.com/watch?v=bQDkuF6-py4, Tommi’s presentation in London Clojurians. This is a great presentation. The presentation does a great job explaining the differences between different data validation libraries and the various ways you can utilize the malli library for data validation.#2020-12-1008:26Kari MarttilaI found particularly interesting the beginning of the presentation. Tommi does a good job visualizing how Plumatic Schema, Spec and Malli specify an example data.#2020-12-1008:44borkdudeAre the slides available publically?#2020-12-1008:47dharriganThat was a great talk btw. I use malli (with Reitit) a lot!#2020-12-1009:03ikitommithanks!#2020-12-1009:03ikitommislides are here: https://www2.slideshare.net/metosin/designing-with-malli#2020-12-1401:11rutledgepaulvI am a little confused by map entries in malli. I see that i can use malli.core/entries to get k/v tuples where the v is a reified :malli.core/val schema that contains the entry's properties, but i sort of expected the map entry itself to also be a reified schema node that I could interrogate properties from. For now I am listing all entries and seeking to find the one that matches my key since malli.util/get returns the value not wrapped in a :malli.core/val and so is missing entry properties#2020-12-1401:11rutledgepaulvIt might be nice if malli.util contained a "find" like "clojure.core/find" for getting a map entry given a key in constant time#2020-12-1401:12rutledgepaulvam i missing something? i'm sure map entries are some of the clunkier things to deal with#2020-12-1401:13rutledgepaulvMaybe what i am missing is that my property is really about the "value" and not about the map entry and so i put it in the wrong place.#2020-12-1911:13raymcdermottHey @ikitommi we recently had @dominicm on defn and he took issue with my pronunciation of Malli. But just listened to your excellent talk at re:Clojure and your pronunciation matches my own. He thinks you’re wrong 😂#2020-12-1911:14raymcdermottIs it your Tampere accent?#2020-12-1911:15dominicmOh no, ousted! I don't even remember disagreeing. Maybe I just like contradicting you @raymcdermott#2020-12-1911:15raymcdermottOh no you don’t #2020-12-1911:15raymcdermott[ sorry it’s Pantomime season ]#2020-12-1911:16raymcdermottAnyway we’ll let you get on with more interesting topics :)#2020-12-1911:22borkdudeBtw, I enjoyed the podcast, but I'm now also aware of how dominicm manipulated me and Alex to do stuff for him!#2020-12-2114:33ikitommi@raymcdermott listened that defn, good stuff. Finnish is simple, just pronounce the way you write it, right? 😉#2020-12-2114:35ikitommibut, reitit is pronounced ray-tit.#2020-12-2117:26dominicmCan confirm, Ray is a tit ;) @raymcdermott#2020-12-2114:36ikitommi@rutledgepaulv the entry handling is bit too hard atm, but not sure what woud be a simpler way.#2020-12-2114:36ikitommithere is a way to ask for the entry instead of the value, just a sec.#2020-12-2115:31ikitommioh, undocumeted, untested and had a bug, but maybe something like this?
(mu/find
  [:map [:b {:optional true} int?]]
  :b)
; => [:b {:optional true} int?]

(-> [:map [:a [:map [:b {:optional true} int?]]]]
    (mu/get :a)
    (mu/find :b))
; => [:b {:optional true} int?]
#2020-12-2115:33ikitommiwhere mu/find is just sugar on top of the hidden feature of “give me the entry tuple” of:
(-> [:map [:a [:map [:b {:optional true} int?]]]]
    (mu/get-in [:a [::m/find :b]]))
; => [:b {:optional true} int?]
#2020-12-2116:12ikitommihttps://github.com/metosin/malli/pull/322#2020-12-2116:13ikitommibtw, those interested in the sequence schemas, here’s the PR (draft): https://github.com/metosin/malli/pull/317#2020-12-2116:17ikitommi
(m/explain
  [:+ [:cat*
       [:prop string?]
       [:val [:alt* [:s string?] [:b boolean?]]]]]
  ["-server" :foo "-verbose" true "-user" "joe"])
=>
{:schema [:+ [:cat* [:prop string?] [:val [:alt* [:s string?] [:b boolean?]]]]],
 :value ["-server" :foo "-verbose" true "-user" "joe"],
 :errors (#Error{:path [0 :val :s], :in [1], :schema string?, :value :foo}
          #Error{:path [0 :val :b], :in [1], :schema boolean?, :value :foo})}
#2020-12-2222:41elaroussHi, Is it possible to write a schema for this vector:
[{:type :work
  :value "
the vector should only contain those 2 items, where the type is kind of static, the value can change. i thought about using this schema:
[:vector [:map [:type keyword?] [:value string?]]]
but it’s not enough for my use case, because i want when i inspect the schema to know how many items i can have in the vector, and their types (the :type ) is this possible? Thank you for the great lib 🙏
#2020-12-2223:14alpox@U6AE62UCT I can think of two ways:
(def data-schema [:map
                   [:type [:enum :work :personal]]
                   [:value string?]])
for the maps and then
(def schema [:vector {:min 2 :max 2} data-schema])
or maybe
(def schema [:tuple data-schema data-schema])
for putting together the schema for the vector
#2020-12-2307:12elaroussI was thinking about something like
[:vector
     [:map [:type :work] [:value string?]]
     [:map [:type :personal] [:value string?]]]
but it didn’t work Thank you @U6JS7B99S i’ll probably go with your first suggestion
#2020-12-2308:08ikitommiif the :type defines the shape of the data, you can try :multi:
(require '[malli.generator :as mg])

(mg/sample
  [:vector {:min 2, :max 2}
   [:multi {:dispatch :type}
    [:work [:map [:type [:= :work]] [:value string?]]]
    [:personal [:map [:type [:= :personal]] [:value string?]]]]])

(mg/sample
  [:tuple {:registry
           {::element
            [:multi {:dispatch :type}
             [:work [:map [:type [:= :work]] [:value string?]]]
             [:personal [:map [:type [:= :personal]] [:value string?]]]]}}
   ::element ::element])
#2020-12-2308:09ikitommiboth emit results like:
([{:type :personal, :value ""} {:type :work, :value ""}]
 [{:type :work, :value "l"} {:type :personal, :value ""}]
 [{:type :personal, :value "z1"} {:type :personal, :value ""}]
 [{:type :personal, :value "ZG"} {:type :personal, :value "N"}]
 [{:type :personal, :value "qQ6"} {:type :work, :value "A8h"}]
 [{:type :personal, :value "2588"} {:type :personal, :value "Djv12"}]
 [{:type :work, :value ""} {:type :personal, :value "c"}]
 [{:type :work, :value "U2uHMI"} {:type :personal, :value "376ihr"}]
 [{:type :personal, :value "n6Ct4d"} {:type :personal, :value "V09xL8"}]
 [{:type :work, :value "9"} {:type :personal, :value ""}])
#2020-12-2308:10ikitommiif the first map is always work and second personal, :tuple for the win:
[:tuple
 [:map [:type :work] [:value string?]]
 [:map [:type :personal] [:value string?]]]
#2020-12-2308:22elaroussThank you, the :multi works for my use case 🙏#2020-12-2308:08ikitommiif the :type defines the shape of the data, you can try :multi:
(require '[malli.generator :as mg])

(mg/sample
  [:vector {:min 2, :max 2}
   [:multi {:dispatch :type}
    [:work [:map [:type [:= :work]] [:value string?]]]
    [:personal [:map [:type [:= :personal]] [:value string?]]]]])

(mg/sample
  [:tuple {:registry
           {::element
            [:multi {:dispatch :type}
             [:work [:map [:type [:= :work]] [:value string?]]]
             [:personal [:map [:type [:= :personal]] [:value string?]]]]}}
   ::element ::element])
#2020-12-2308:42borkdude@ikitommi In other languages people usually define this as an ADT. Maybe :enum can also be used for this?#2020-12-2308:43borkdude(I didn't see any docs about :enum in the README, only examples)#2020-12-2308:43ikitommi:enum is about values, :or is about schemas:
[:or
 [:map [:type [:= :work]] [:value string?]]
 [:map [:type [:= :personal]] [:value string?]]]
fixed, thanks to @borkdude
#2020-12-2308:44borkduderight, that's what I was thinking#2020-12-2308:44ikitommibut, :or does a linear scan over schema, :multi can be much more efficient.#2020-12-2308:45borkdudewhy did you have to write [:work [:map [:type [:= :work]] [:value string?]]] with :multi but [:map [:type :work] [:value string?]] with :or?#2020-12-2308:45ikitommi(`either` was deprecated in Plumatic Schema because of this, malli supports that as it’s usefull to describe the real world, despite being slow)#2020-12-2308:46ikitommicould be:
[:work [:map [:type string?] [:value string?]]]
#2020-12-2308:46borkdudeif you support :case for a closed world of possibilities it could possibly be even faster?#2020-12-2308:46ikitommi… but the generator would then just generate any keyword for the :a: #2020-12-2308:47ikitommiwhat kind of code would the :case emit?#2020-12-2308:47borkdudeI was referring to [:type [:= work]] vs [:type :work]#2020-12-2308:47ikitommioh, that’s my bug. [:type :work] doesn’t work.#2020-12-2308:47ikitommilooks for :work from the registry, which doesn’t exist.#2020-12-2308:48borkdudeah ok then#2020-12-2308:48borkdude(I have no idea what :case would emit, just bouncing an idea)#2020-12-2308:48ikitommi[:= :work] is basically same as [:enum :work]#2020-12-2308:48borkdudeso [:map [:enum :work :personal] [:value string?]] would also work#2020-12-2308:49ikitommi[:map [:type [:enum :work :personal]] [:value string?]] yes#2020-12-2308:50ikitommibut if the :type effects the :value , or any other parts of the schema, then :multi is the way.#2020-12-2308:51borkdudedoes it generate a multi-method behind the scenes?#2020-12-2308:51ikitommino, it’s closed and immutable by design.#2020-12-2308:52ikitommijust a dispatch map from key -> schema. and the key is looked up using the :dispatch function.#2020-12-2308:52ikitommi:dispatch function is applied to value, which returns the :multi key, which selects the schema#2020-12-2308:53ikitommiand, :dispatch can be a sci-function, of course 🙂#2020-12-2308:53ikitommi
(m/validate
  [:multi {:dispatch 'first}
   [:sized [:tuple keyword? [:map [:size int?]]]]
   [:human [:tuple keyword? [:map [:name string?] [:address [:map [:country keyword?]]]]]]]
  [:human {:name "seppo", :address {:country :sweden}}])
; true
#2020-12-2308:54ikitommiso, there is always a map-lookup, so not as fast as case, but it would require code-generation, which would mean a macro.#2020-12-2308:55borkdudeyeah, that makes sense#2020-12-2308:57ikitommicould add a :conditional, where the entry keys are functions, and the first one will match. Like :or, but more explicit and short-circuits on first match:
[:conditional
 [map? [:map [:x int?]]]
 [int? [:int {:min 1, :max 2}]]]
#2020-12-2308:58ikitommimalli.core is distilled into tons of helper functions to build new schemas, really easy to add things like this atm.#2020-12-2308:59borkdudedoesn't :or short-circuit?#2020-12-2309:01borkdudebtw, I also implemented case as a map-lookup in sci, same problem ;)#2020-12-2309:04ikitommisorry, it does. I think the issue is that you can have multiple branches that would match, making the rest effectively unreachable.#2020-12-2309:06ikitommialso, the :or tranformation is bit hacky, depending whether you are doing decoding or encoding, we need to check if the transformation turned the value valid or not. that’s slow.#2020-12-2309:06ikitommi… but, works 🙂#2020-12-2602:23steveb8nI’ve started switching out Spec for Malli to improve performance in hot code. it works great except when using recursive schemas. I’ve created a gist to illustrate https://gist.github.com/stevebuik/e4f3475e46dd5ebb1de7707438fa073f#2020-12-2602:24steveb8nanyone seen this before? Or am I making a mistake somehow?#2020-12-2609:56ikitommi@steveb8n you are using :and, which composes the two maps - in validation it runs two validations, one for each map. If you use malli.util/merge, the end result is one schema, and validation runs it just once. tested the first example, it’s 90ns vs 97ns, which is the cost of iterating over 2 MapEntrys instead of just one.
(require '[malli.util :as mu])

(def simple-tree-node
  (mu/merge
    simple-node
    [:map {:registry {::child simple-node}}
     [:children {:optional true} [:vector [:ref ::child]]]]))
#2020-12-2611:35steveb8nNow that's a Xmas gift! I'll try it tomorrow. Thanks#2020-12-2610:01ikitommione note about merge: both the properties and children are merged (no deep merge atm), so the last :registry property wins. hadn’t thought of that :thinking_face:#2020-12-2610:01ikitommibut, here, it’s:
simple-tree-node
;[:map
; {:registry #:recursive-malli-perf{:child [:map [:name string?]]}}
; [:name string?]
; [:children {:optional true} [:vector [:ref :recursive-malli-perf/child]]]]
#2021-12-2702:43euccastrosorry if I missed something basic, but is there a way to specify that a string field should match a regular expression, without resorting to SCI functions?#2021-12-2707:51steveb8nhave you considered using :fn without using sci? why is that not a good solution?#2021-12-2713:20ikitommihttps://github.com/metosin/malli/blob/master/test/malli/core_test.cljc#L484-L509#2021-12-2713:20ikitommiany documentation PRs most welcome#2021-12-2715:23euccastroPR here: https://github.com/metosin/malli/pull/323/files#2021-12-2715:41ikitommithanks!#2021-12-2704:46steveb8n@ikitommi can I trouble you for one more gist comment? this is not urgent so I’m happy to wait until your status is not “vacationing” 🙂 https://gist.github.com/stevebuik/e63735d99fca94041120f9b0e25b616d#2021-12-2705:05steveb8nnot urgent because I can work around the perf by not using recursion in my initial spec replacements. full recursion will be useful but can come later#2021-12-2705:05steveb8nand because OSS should never be urgent 🙂#2021-12-2706:23steveb8nI wonder if I/we could turn this into a useful sample for others to learn from? happy to do this if you agree it would be useful#2021-12-2713:24ikitommithanks for the awesome repro, easy to dig in to this when have the extra time.#2021-12-2801:24steveb8nThank you for such a big upgrade to Spec 🙂 BTW I’m also noticing some very slow perf when compiling medium complexity nested schemas. Are you interested in test cases for that as well?#2021-12-2804:42steveb8nThe compilation slowness appears to be caused by liberal use of :merge. when I remove it and just compose maps (i.e. inline) I see compile speeds improve by 10x#2021-12-2805:03steveb8nI’ve worked around both the compilation (i.e. validator) and runtime perf issues so this is not urgent for me. I’ll be happy to provide more test cases if you need them#2021-12-2718:27borkdudeHow does malli expect bytes? to be delivered e.g. when you have a JSON api#2021-12-2718:27borkdudeas base64? (that would seem random)#2021-12-2718:28borkdudeor would one do a pass over the json, deserialize the base64 manually?#2021-12-2718:28borkdudeI'm asking this since babashka pods communicate via edn or json, which doesn't have a built-in way to represent bytes. transit does have it#2021-12-2719:27ikitommi@borkdude atm, there is no default encoding/decoding bytes and strings. Looking at OpenAPI docs, the guide is: > base64-encoded characters, for example, U3dhZ2dlciByb2Nrcw==#2021-12-2719:28ikitommihttps://swagger.io/docs/specification/data-models/data-types/#2021-12-2719:28borkdudefor now I went with transit. it does it for me.. I should look into how it does it#2021-12-2719:30borkdude#2021-12-2719:29ikitommiso, there could be a byte<->string transformers in malli.transform to make this a default#2021-12-2723:12steveb8nIs this a bug? If so, I will log it…. (m/validator [:schema {:registry (merge (m/base-schemas) {:person [:multi {:dispatch :person/gender}]})} :person]) Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79). :malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1}#2021-12-2723:16steveb8nworkaround….#2021-12-2723:17steveb8n
(m/validator [:schema
              {:registry {:person [:multi {:dispatch :person/gender}]}}
              :person])
#2021-12-2916:16aaron51How do I describe keywords with a certain namespace? Something like [:qualified-keyword {:namespace :asdf}] to match :asdf/x and :asdf/y#2021-12-2916:55ikitommi@aaron51 something like [:and :qualified-keyword [:fn (fn [x] (= :asdf (namespace x))]]?#2021-12-2916:56aaron51yes! thank you#2021-12-2916:58ikitommiYou can also create your own :qualified-keyword impl quite easily, on top of malli.core/-simple-schema, which would read the :namespace property and validate it behind the scenes. e.g. the syntax you proposed. Not near computer, but there are examples how to do that kind of stuff in the user space.#2021-12-2916:59ikitommiAlso, sounds useful enough, so you could do a PR of that.#2021-12-2917:00aaron51thank you, I’ll give it a try 👍#2021-12-2917:04ikitommiSome examples how to build custom s (simple) schemas: https://github.com/metosin/malli/blob/master/test/malli/core_test.cljc#L1392-L1470#2021-12-3018:56Michael WIs it possible to take a swagger file and generate a malli schema? I see how to convert schema to swagger, but not the reverse in the docs.#2021-12-3020:23ikitommi@michael819 currently, no. There is a stalled PR of JSON Schema -> malli conversion. I would guess it takes days / weeks to make it complete enough, don't have the time myself for that. But, contributions most welcome.#2021-12-3020:44Michael WI think it's a chicken and egg problem. Should I create a malli schema for openapi to validate the files, then work from there? Just trying to gauge the amount of work necessary and a general idea of where to start.#2021-12-3120:17eoliphantHi, I’m using malli ‘spec’ishly’ with the mutable custom schema suggested in the readme. One annoyance I’m running into is the need to set the malli.registry/type to custom as a JVM property in quite a few places. I’n the current case, I’m running a Fulcro app as a Datomic ion. I ended up needing to set the property to run shadow-cljs itself (not just setting it as a closure define), the ion cmd line tools, etc, and I’m calling System/setProperty directly in code in the server entry points to try to make sure it’s set there as you can’t really set system props for the server runtime. Just wondering if longer term there might be a better way to handle this#2021-01-0116:57ikitommi@eoliphant the first alpha wanted to be immutable by default. We could revisit that for 1.0.0, starting to think it would be ok if malli was optionally immutable: the JVM/compiler option could be to close things up.#2021-01-0117:01ikitommiwhen one does multi-tenant schema systems, anyway the registry is either local or passed as option -> one can have already full control of the registries in places where it matters. The system default registry is just for your own core app, not the tenants.#2021-01-0117:01ikitommimaybe have a up-to-date branch of malli with the option reversed?#2021-01-0117:04ikitommialso, comments welcome on having an optional global registry. Something like (malli.mutable/register! :user/name :string)#2021-01-0202:35y.khmelevskiiHi! I would like to ask about feature request for malli.generator. It would be great to support relations and constrains like in https://github.com/reifyhealth/specmonstah For example
(def schema
  {:user {:prefix :u
          :spec   ::user}
   :post {:prefix    :p
          :spec      ::post
          :relations {:created-by-id [:user :id]}}
   :like {:prefix      :l
          :spec        ::like
          :relations   {:post-id       [:post :id]
                        :created-by-id [:user :id]}
          :constraints {:created-by-id #{:uniq}}}})
Is it make sense for malli roadmap?
#2021-01-0209:41ikitommi@U1GTUPAVB specmonstah does a lot of things and it might be generic enough to support malli easily. Would you like to investigate / suggest that to specmonstah maintainers?#2021-01-0209:44ikitommibut, please write the issue to malli repo, so it's on potential backlog. The idea is good, would use that myself.#2021-01-0209:45ikitommiRelated: https://github.com/metosin/malli/issues/53#2021-01-0622:37steveb8nI use specmonstah so I'm interested in this issue as well#2021-01-0700:12pithylessFYI - There is a fork of specmonstah meant to support malli, but I haven’t been tracking it so not sure how baked it is: https://github.com/lambdaisland/specmonstah-malli#2021-01-0700:19steveb8nmaybe we should suggest a name change for the fork 🙂#2021-01-0611:20ordnungswidrigGive a definition like
(def Foo [:schema {:registry {"Foo" [:map [:bar pos-int?}  "Foo"])
Is there a way to “resolve” that schema to the actual schema referenced by “Foo”? I want to programatically access the actual definition of “Foo”, e.g. to enumerate the map fields. I’ve look through the properties, options, children and reference of the schema created by (m/schema Foo) but I’m a little lost.
#2021-01-0617:01ikitommi@ordnungswidrig try (m/deref Foo)#2021-01-0617:02ikitommithere is also m/deref-all, which peels all top-level RefSchemas.#2021-01-0617:04ikitommialso, you can always m/walk the schema with option :malli.core/walk-refs, which walks over all refs. Not sure of that one blows up with recursive schemas...#2021-01-0617:39ordnungswidrigOh, i had tried ref but not deref.#2021-01-1115:11hansbuggeNone of the comparator schemas have default error messages, e.g.,
(-> (m/explain [:> 0] -1)
    :errors first me/error-message)
;; => "unknown error"
Is there a good reason why some simple error functions haven't been added to malli.error/default-errors other than "no-one has done it yet"? 🙂
#2021-01-1115:40hansbuggeI've opened a PR with suggestions for some simple error functions https://github.com/metosin/malli/pull/332#2021-01-1118:40ikitommiMerged the PR, thanks!#2021-01-1518:47ikitommi🚀 🚀 🚀 regex schemas merged in master! feel free to test: validate, explain, transform and generate - parsing/conform still wip.#2021-01-1518:55ikitommithanks to @nilern, the impl is quite snappy, currently order(s) of magnitude faster than spec & seqexp.#2021-01-1700:04lreadI’m trying out malli for the first time. I’m using it to validate user entered options. I really like the humanize spell check support. Check out this helpful error message!
Invalid inline options: #:test-doc-blocks{:read-cond ["should be spelled :test-doc-blocks/reader-cond"]}
 file: doc/design/namespaced-elements.adoc line: 277
#2021-01-1700:04lreadAnyhoo, thanks!#2021-01-1718:50naomarikI've yet to update malli since before it had an actual version number. Looks way more feature rich now.#2021-01-1719:13ikitommithe function schemas are in master, but not released. there is a bug with generating multi-arity functions and as the sequence/regex schemas are in, will add support for varags. Then good to release a new version. Hopefully much more and smaller releases after that.#2021-01-1719:14ikitommibtw, comments welcome on the malli.error/humanize output: few options how to fix that: https://github.com/metosin/malli/pull/333#discussion_r559224770#2021-01-1819:48Samuel McHughHello, I'm trying out Malli and I'm wondering what is the idiomatic approach to spec'ing a finit explicit set of values. Here's an example of a schema which works.
(m/validate
  [:or [:= :foo]
       [:= :bar]
       [:= :baz]]
  :foo)
;=> true
The syntax isn't as compact as I'd expect though. I'd hope something like
(m/validate
  #{:foo :bar :baz}
  :foo)
could work but I haven't seen this in the docs anywhere. Do I have to set up a custom registry for this?
#2021-01-1820:02ikitommi@smchugh230395 try [:enum :foo :bar :baz]. Doc PRs welcome!#2021-01-1909:06Samuel McHughI will have a look after work today and make a doc PR. Thanks!#2021-01-1917:56Samuel McHughI'm going to spend more time understanding what's available and possible before I submit any doc changes. There is a lot to chew on! Is there a native capacity to enforce uniqueness of items in a collection?#2021-01-2009:18ordnungswidrig@smchugh230395 can you use a set as the collection type?#2021-01-2009:20Samuel McHughThat's a good idea but there is an order on the elements of the collection in my particular use-case. I can set up other ways of ensuring that only unique items enter the collection though so it isn't critical that Malli supports it out of the box.#2021-01-2009:46ordnungswidrigI see. So “ordered set” is what you need I guess.#2021-01-2009:52ordnungswidrig(m/validate [:and [:sequential any?] (fn [c] (= (count c) (count (set c)))) something like this?#2021-01-2014:22Samuel McHughgood one, I think i'll go with this#2021-01-2014:51ordnungswidrigPretty sure this one of the least optimized way to check for distinct elements (reduce (fn [xs x] (if (xs s) (reduced false) (conj xs s))) #{} coll) might be faster.#2021-01-2010:46ikitommiJSON Schema has “uniqueItems” attribute, I guess sequences could have :distict boolean property?#2021-01-2010:46ikitommihttps://json-schema.org/understanding-json-schema/reference/array.html#uniqueness#2021-01-2010:47ikitommi… would be [:sequential {:distinct true} any?]#2021-01-2011:32ordnungswidriginteresting#2021-01-2014:09emccueI am somewhat meh on the coupling to json schema#2021-01-2014:10emccueis there some usecase or justification i am missing?#2021-01-2119:06Daniel MiladinovHi everyone, long-time listener, first-time caller here. I’m still kind of new to malli and I’m trying to develop a spec to validate a vector of strings, but some of the strings have different validations than the others. Looking at the https://github.com/metosin/malli#sequence-schemas, it seemed to me that I should use a sequence schema with something like [:cat …].#2021-01-2119:07Daniel MiladinovBut I’m having a problem. According to the README, I should be able to do this:
(m/validate [:cat string? int?] ["foo" 0]) ; => true
But in my repl, I get this:
(require '[malli.core :as m])
=> nil
(m/validate [:cat string? int?] ["foo" 0])
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :cat}
#2021-01-2119:23Daniel MiladinovThe project I’m working in has reitit 0.5.11 as a direct dependency, which appears to pull in malli 0.2.1 as a transitive dependency.#2021-01-2120:02ikitommiif you use deps, you can take a dependency to the latest commit of Malli, which has the sequence schemas. Will try to ship a Big 0.3.0 out within few weeks with those officially in.#2021-01-2120:03ikitommiThere is also [:tuple string? int?] for fixed length vectors with schema for each element#2021-01-2120:03Daniel MiladinovThank you!!!#2021-01-2120:03Daniel MiladinovThat will serve my purposes in the meantime#2021-01-2315:14ikitommim/conform landed in master by @nilern! #343 will change the names to m/parse and m/parser:
(m/parse
  [:* [:cat*
       [:prop string?]
       [:val [:alt*
              [:s string?]
              [:b boolean?]]]]]
  ["-server" "foo" "-verbose" true "-user" "joe"])
;[{:prop "-server", :val [:s "foo"]}
; {:prop "-verbose", :val [:b true]}
; {:prop "-user", :val [:s "joe"]}]
#2021-01-2410:15ikitommiby the power of generic schema walking (answer to #343):
(require '[malli.core :as m])

(defn normalize-properties [?schema]
  (m/walk
    ?schema
    (fn [schema _ children _]
      (if (vector? (m/form schema))
        (into [(m/type schema) (m/properties schema)] children)
        (m/form schema)))))

(normalize-properties
  [:map
   [:x int?]
   [:y [:tuple int? int?]]
   [:z [:set [:map [:x [:enum 1 2 3]]]]]])
;[:map nil
; [:x nil int?]
; [:y nil [:tuple nil int? int?]]
; [:z nil [:set nil
;          [:map nil
;           [:x nil [:enum nil 1 2 3]]]]]]
#2021-01-2510:43ikitommim/parse and m/parser now in master. Feedback and test reports most welcome.#2021-01-2514:45kwrooijenm/parse looks great! I think that's going to clean up a lot of my code once I get a chance to use it#2021-01-2518:34ikitommiupdated README with parsing examples: https://github.com/metosin/malli#parsing-values#2021-01-2518:36ikitommithe hiccup example is from minimallist, thanks to @U8MJBRSR5 for the original example 🙂#2021-01-2518:39Vincent CantinI like it 🙂#2021-01-2518:56Vincent Cantin@U055NJ5CC in Malli’s version, there is no indication that the :node is a vector. Was it intentional?#2021-01-2519:23ikitommiOh, that is missing. It accepts all sequences#2021-01-2519:24ikitommican't compose with :and here, need to constraint in some other way#2021-01-2519:35Vincent Cantinwithout the vector indication, the generator might create a list, that's a problem.#2021-01-2519:37ikitommiyou can add :gen/fmap vec to gen a vec, but having a :kind etc. property would make it work in all the places (validate, gen, transform)#2021-01-2518:35ikitommi:or, :cat and :alt all use non-named branches, :or* , :cat* and :alt* are the named variants.#2021-01-2518:36ikitommiif somenone has better names for those, please suggest.#2021-01-2518:42Vincent CantinNaming things is hard, indeed.#2021-01-2519:09oliverHi! I'm using malli for the first time on a project where I have to derive HTML-Forms from the body-structure of http-requests. So far the experience has been great – and much more natural than with spec. Now I feel the need for something I was unable to find in the Readme. Say I have the following schema definition: [:map {:closed true} [:a int? :b int?]] I would like to further constrain :b depending on :a, e.g.: - :b must be greater than :a — or — - :b should only be present iff :a is greater than 3 (nil otherwise) Is there a way to express this without recourse to [:fn ] schemas? I'll be thankful for any hints on this!#2021-01-2520:56emccuehow would I annotate a function for runtime checking at dev time?#2021-01-2521:19ikitommi@emccue no such thing yet, but should be easy to add. contributions welcome. https://github.com/metosin/malli/issues/349#2021-01-2521:20ikitommi@services :and & :fn is the way to do it now.#2021-01-2521:21ikitommihttps://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&amp;schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D#2021-01-2717:59oliverHi, sorry for taking so long to respond… many thanks for the playground example… I wasn't even aware that existed. I'll be using [:fn …] then. straightforward!#2021-01-2812:50danielnealIs there a way of converting malli definitions to swagger 3 format?#2021-01-2815:46juhoteperiNot sure about Swagger 3, malli.swagger might be do Swagger 2, but not sure if there is difference in this level. It is just JSON Schema with a few extensions.#2021-01-2816:51danielnealI've been reading the swagger specification and it's massive#2021-01-2816:51danielnealhttps://swagger.io/specification/#2021-01-2816:53danielnealThe most obvious difference is that in JSON Schema and Swagger 2, there is a "definitions" key for common objects, whereas this doesn't exist in OpenAPI 3#2021-01-2816:54danielnealI think it's replaced by the components object, or the schemas property on the components object#2021-01-2816:56juhoteperiWith currently implementation, Malli only handles the schema -> properties part. Reitit/Ring-swagger/Compojure-api does the higher level Swagger2 generation.#2021-01-2816:57juhoteperiThe properties part (JSON Schema) probably doesn't change much if at all between Swagger2 / OpenAPI 3.#2021-01-2817:07danielnealdo you know if there are plans to support OpenAPI 3 in reitit?#2021-01-2817:08danielnealah yeah, there's an open issue https://github.com/metosin/reitit/issues/84#2021-01-2813:00ikitommiJust chatted with @juhoteperi on Metosin internal about that. should be, not atm. should be a thin layer on top of malli.json-schema.#2021-01-2813:01ikitommiwho would like to contribute those?#2021-01-2813:12danielnealhaha great timing#2021-01-2813:39danielnealah oh you mean you chatted with him after my comment 🙂#2021-01-2813:39danielnealI could have a look at it I guess, but swagger and malli are both completely new to me#2021-01-2813:40ikitommiJust = yesterday. https://github.com/metosin/malli/pull/354#2021-01-2815:10ikitommithis is going to be so good.#2021-01-2815:11ikitommi#2021-01-2815:46juhoteperiNot sure about Swagger 3, malli.swagger might be do Swagger 2, but not sure if there is difference in this level. It is just JSON Schema with a few extensions.#2021-01-2910:43danielnealwhat order do :and schemas operate in#2021-01-2910:43danielnealis it possible to do
(m/explain [:and
              int?
              (m/-simple-schema
                {:pred #(> % 3)})] "asdf")
#2021-01-2910:44danielnealand have the int? check prevent the pred from running and erroring?#2021-01-2911:02danielneallike in spec
(s/def ::greater-than-3
    (s/and int? #(> % 3)))
#2021-01-2911:13dharriganThis will work...#2021-01-2911:13dharrigan(def foo [:and int? [:fn (fn [x] (> x 3))]])#2021-01-2911:13dharrigan
#'user/foo
user=> (m/validate foo "abcd")
false
user=> (m/validate foo 1)
false
user=> (m/validate foo 3)
false
user=> (m/validate foo 4)
true
#2021-01-2911:16dharriganYou can, of course, shorten it, i.e., (def foo [:and int? [:fn #(> % 3)]])#2021-01-2911:18danielnealinteresting, I wonder why :fn works but -simple-schema doesn't#2021-01-2911:19dharriganIt's in the documentation to use that 🙂#2021-01-2911:19dharriganYou can try it out here as well #2021-01-2911:26danielnealcool, I'll use :fn 🙂#2021-01-2911:29dharriganActually, it works with simple-schema too#2021-01-2911:30dharrigan
user=> (def foo2 [:and int? (m/-simple-schema {:pred #(> % 3)})])
#'user/foo2
user=> (m/validate foo2 4)
true
user=> (m/validate foo2 3)
false
user=> (m/validate foo2 1)
false
user=> (m/validate foo2 "and")
false
user=> 
#2021-01-2911:30dharriganI missed that, so I suppose you can use that as well 🙂#2021-01-2911:31dharriganchoices...choices...#2021-01-2911:41danielneal@dharrigan, validate succeeds, but explain fails#2021-01-2911:41danielnealcould be a bug perhaps?#2021-01-2911:41danielneal(m/explain foo2 "and") => java.lang.ClassCastException#2021-01-2912:08danielnealI guess it does makes sense that explain on an :and would give all the reasons something fails, but it doesn't make sense -simple-schema fails where :fn doesn't.#2021-01-2912:09juhoteperiIf you are just checking > you could also use [:int {:min 3}] or [:and int? [:>= 3]].#2021-01-2912:09juhoteperiFirst one works best with JSON Schema#2021-01-2912:12danielnealah, the actual example here is a bit different, it's for a kebab case keyword#2021-01-2912:13danielneal
[:and
    {:description "A keyword joined by hyphens"
     :swagger/description "A string joined by hyphens"
     :swagger/example "kebab-case-string"
     :error/message "should be a kebab-case keyword"}
     keyword?
    [:fn kebab-case-keyword?]]
#2021-01-2912:14danielneal(where in this case kebab-case-keyword? fails on non-keywords)#2021-01-2915:08ikitommiWIP: Schemas of Schemas:
(-> (m/-simple-schema
      {:type :int
       :pred string?
       :properties-schema [:map [:min nat-int?] [:max nat-int?]]
       :property-pred (m/-min-max-pred nil)})
    (m/properties-schema))
; => [:map [:min nat-int?] [:max nat-int?]]
also:
(let [OneOf (m/-simple-schema
              (fn [{:keys [count]} values]
                (let [value? (set values)]
                  {:type :user/over
                   :pred value?
                   :properties-schema [:map [:count [:= count]]]
                   :children-schema (into [:tuple] (map (fn [x] [:= x]) values))
                   :type-properties {:error/message (str "should be one of " values)
                                     :json-schema {:type "oneOf", :values values}}})))
      schema (m/schema [OneOf {:count 6} 1 2 3 4 5 6])]
  {:properties-schema (m/properties-schema schema)
   :children-schema (m/children-schema schema)})
;{:properties-schema [:map [:count [:= 6]]]
; :children-schema [:tuple [:= 1] [:= 2] [:= 3] [:= 4] [:= 5] [:= 6]]}
#2021-02-0313:05danielneallol, trolling hard rich#2021-02-0315:52pbailleHi, i'm playing with malli and am confused by this example from the README:
(m/validate [:? int?] [1 2])
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79). :malli.core/invalid-schema {:schema :?}
#2021-02-0316:23ikitommi@pbaille README describes what is in the master, sequence/regex schemas are not in official release. If you take the latest commit as dependency via deps, there are there#2021-02-0316:24ikitommiwill cut a release soon, work needed for the function schemas still#2021-02-0316:31pbaille@ikitommi thank you a lot 🙂#2021-02-0414:27dakraHow can I read the properties of attributes in a map. E.g. the description of one-attr in this example
(def test-schema
    (malli/schema [:map {:description "test schema desc."}
                   [:one-attr {:description "attr description"} string?]]))

  (malli/properties test-schema)  ;; => {:description "test schema desc."}
  
  (malli/properties (mu/get test-schema :one-attr))  ;; => nil
#2021-02-0416:25ikitommi@daniel415 try (mu/find schema key) and you can iterate the child entries with m/children#2021-02-0416:30dakraThe mu/get does work and gives me the correct schema back but just m/properties is not returning anything. (only for maps) And m/children is not working in my example?
(-> (malli/schema [:map {:description "test schema desc."}
                     [:one-attr {:description "attr description"} string?]])
      (mu/find :one-attr)
      malli/children)
return this error
Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:12).
:malli.core/invalid-schema {:schema :one-attr}
#2021-02-0416:32ikitommitry removing the last m/children#2021-02-0416:33ikitommimu/find should return the entry tuple, props being the second value#2021-02-0416:35ikitommialso, this should hold:
(-> [:map [:x [:int {:default 1}]]]
    (m/get :x)
    (m/properties))
;; -> {:default 1}
#2021-02-0416:35ikitommicoding blind from a phone, might contain errors :)#2021-02-0416:36ikitommibut: maps, map entries and map entry values can all have properties#2021-02-0416:37dakraHmm, your example works. But when I simply have string? or so it doesn't work.
(-> [:map [:x {:default 1} string?]]
      (mu/get :x)
      (m/properties))
  ;; => nil
#2021-02-0416:38dakraSo this is the equivalent that works:
(-> [:map [:x [:string {:default 1}]]]
      (mu/get :x)
      (m/properties))
  ;; => {:default 1}
#2021-02-0418:52ikitommistring? doesn’t have properties. is is short format for [string?]. You can add properties to it too, e.g. [string? {:default 1}]. Full example:
(-> [:map [:x [string? {:default 1}]]]
      (mu/get :x)
      (m/properties))
  ;; => {:default 1}
(still not at malli repl, just guessing what it returns)
#2021-02-0418:58ikitomminow at the repl:
(def Schema
  [:map {:in "map"}
   [:x {:in "entry"} [:string {:in "value"}]]])

(m/properties Schema)
;; => {:in "map"}

(-> Schema
    (mu/find :x)
    (second))
;; => {:in "entry"}

(-> Schema
    (mu/get :x)
    (m/properties))
;; => {:in "value"}'
#2021-02-0419:00ikitommisome helpers:
(defn entry-properties [schema key]
  (-> schema (mu/find key) (second)))

(defn value-properties [schema key]
  (-> schema (mu/get key) (m/properties)))

(m/properties Schema) ;; => {:in "map"}
(entry-properties Schema :x) ;; => {:in "entry"}
(value-properties Schema :x) ;; => {:in "value}
#2021-02-0419:44dakraOK, thank you very much. Now it's clear. I wasn't aware that string? is shorthand for [string?] and also didn't really think about the difference between entry and value properties. Makes sense now 🙂#2021-02-0609:45ikitommim/unparse and m/unparser landed in master! parse + unparse with hiccup:
(def Hiccup
  [:schema
   {:registry {"hiccup" [:or*
                         [:node [:cat*
                                 [:name keyword?]
                                 [:props [:? [:map-of keyword? any?]]]
                                 [:children [:* [:schema [:ref "hiccup"]]]]]]
                         [:primitive [:or*
                                      [:nil nil?]
                                      [:boolean boolean?]
                                      [:number number?]
                                      [:text string?]]]]}}
   "hiccup"])

(m/parse
  Hiccup
  [:div {:class [:foo :bar]}
   [:p "Hello, world of data"]])
;[:node
; {:name :div,
;  :props {:class [:foo :bar]},
;  :children [[:node
;              {:name :p
;               :props nil
;               :children [[:primitive [:text "Hello, world of data"]]]}]]}]


(->> [:div {:class [:foo :bar]}
      [:p "Hello, world of data"]]
     (m/parse Hiccup)
     (m/unparse Hiccup))
;[:div {:class [:foo :bar]}
; [:p "Hello, world of data"]]
#2021-02-0609:49ikitommitwo small things for 0.3.0 with all the stuff in.#2021-02-0610:20borkdudenice!#2021-02-0610:20borkdudethis is like conform right?#2021-02-0610:40ikitommiyes, same as s/confom and s/unform. At the moment, there are no property-based custom parsers/unparsers that a user could add (like custom property based transformers) and in case of error, just ::m/invalid is returned. I think we can later combine -parse, -unparse and -explain implementations together, so that one can see partially parsed results and get more details when parsing/unparsing fails (same output as in -explain).#2021-02-0708:41ikitommiNot sure how useful the function schema checking is in real life (maybe for tooling?), but here’s the wip:
(require '[malli.generator :as mg])

(def check
  (mg/function-checker
    [:function
     [1 [:=> [:cat :int] :int]]
     [2 [:=> [:cat :int :int] [:int {:max 10}]]]]))

(check
  (fn
    ([x] x)
    ([x y] (mod (+ x y) 10))))
; => nil

(check
  (fn
    ([x] x)
    ([x y] (+ x y))))
;({:total-nodes-visited 18,
;  :depth 6,
;  :pass? false,
;  :result false,
;  :result-data nil,
;  :time-shrinking-ms 0,
;  :smallest [(0 11)],
;  :malli.core/schema [:=> [:cat :int :int] [:int {:max 10}]]})

(check
  (fn
    ([x y] (mod (+ x y) 10))))
;({:total-nodes-visited 0,
;  :depth 0,
;  :pass? false,
;  :result "Wrong number of args (1) passed to: user/eval97462/fn--97463",
;  :time-shrinking-ms 0,
;  :smallest [(0)],
;  :malli.core/schema [:=> [:cat :int] :int]})
#2021-02-0807:16ikitommithis @UG9U7TPDZ, in master, will be released soon.#2021-02-0709:01caumondhi, I search a little bit long before finding that representing date in malli could be easily done with inst? Is it a recommended approach? I have a doubt as I would expect to find a reference of that in the malli doc which I did not.#2021-02-0711:20ikitommiinst? is ok, but see https://github.com/metosin/malli/issues/49#2021-02-0717:12caumondYes, I saw the PR, it was not completely enlightening. I understood something is missing, some of the proposal, but I did not understand there what is the recommend approach awaiting for that PR to be merged.#2021-02-0717:13caumondI keep the "`inst` is ok" part !#2021-02-0711:22ikitommi:not merged in master:
(mg/sample [:not :string])
;([]
; #{}
; nil
; \$
; nil
; nil
; {#uuid"3565d7f3-5561-439f-9368-5bfdbd03fc8e" -2/3}
; ()
; [\^]
; [[-110317951337N \:] #{2.15625 R+3/qfp}])
#2021-02-0807:10kwrooijenHow do you spec a function? e.g. I'd like to do this:
[:map
 [:component/foo 'fn?]]
#2021-02-0807:10kwrooijenBut that results in :malli.core/invalid-schema {:schema fn?}#2021-02-0807:14ikitommi@kevin.van.rooijen you can say: [:map [:component/foo [:fn fn?]] or 'fn? to keep it serializable#2021-02-0807:15ikitommimy code sniplet from yesterday show how one can enforce input & output params and arities too.#2021-02-0807:20kwrooijenCool, [:fn 'fn] works like a charm#2021-02-0807:20kwrooijenThanks, looking forward to the next release 🙂 Will be replacing all my specs with Malli once that happens#2021-02-0813:41ikitomminice. would be great if some non-metosinian blogged about the goods and bads of spec, schema and malli - from real life experience. happy to try fix all the bads in malli 😉#2021-02-0820:01kwrooijenI might do that eventually, although I have a hard time thinking of bad things about Malli, so it'll probably only be success stories 🙂#2021-02-0820:03kwrooijenI'm currently using it for Gungnir, and working on integrating it for Duct / some web development stuff. At my job we heavily use Malli as well (As well as serialized through an API)#2021-02-0820:04kwrooijenI might use the new parser for a (hiccup) templating language that I'm building. So that's pretty cool timing of the new feature 😄#2021-02-0911:48danielnealWhat's the best way to define a string format so that it comes through to swagger#2021-02-0911:49danielneal
(swagger/transform [:map
                    [:a {:swagger/format "date-time"} string?]])
; => {:type "object", :properties {:a {:type "string", :format "date-time"}}, :required [:a]}
#2021-02-0911:49danielnealthis works, but I'm wondering if there's a better way that also has validation#2021-02-0911:51ikitommimaybe split it:
(def DateTimeString [:string {:swagger/format "date-time"}])

(swagger/transform [:map [:a DateTimeString]])
; => {:type "object", :properties {:a {:type "string", :format "date-time"}}, :required [:a]}
#2021-02-0911:55ikitommiwhat do you mean by “also has validation”?#2021-02-0911:56danielneallike maybe a [:fn #(java.time.Instant/parse %)] or something?#2021-02-0911:57ikitommithat would return an Instant, not string.#2021-02-0911:59ikitommiafter the date schemas are implemented, it would be:
(swagger/transform [:map [:a :instant]])
; => {:type "object", :properties {:a {:type "string", :format "date-time"}}, :required [:a]}
#2021-02-0911:59danielnealah, that would be ideal#2021-02-0912:00ikitommibut, do you want Instant as a result or just a string that is formatted like an instant?#2021-02-0912:04danielnealhaha that's a good question. For consumers of the api, and in the docs etc it should be a string that is formatted like an instant. It would be nice if when we coerce parameters, we could get back a real instant.#2021-02-0912:04danielnealIs what :string/decode is used for#2021-02-0913:53ikitommi@danieleneal if you want a full custom type, here’s a sample:
(ns demo
  (:require [malli.core :as m]
            [malli.error :as me]
            [malli.generator :as mg]
            [malli.transform :as mt]
            [malli.json-schema :as json-schema]
            [clojure.test.check.generators :as gen])
  (:import [java.time Instant]
           [java.util Date]))

(def instant
  (let [string->instant #(if (string? %) (Instant/parse %))]
    (m/-simple-schema
      {:type 'instant
       :pred (partial instance? java.time.Instant)
       :type-properties {:error/message "should be instant"
                         :decode/string string->instant
                         :decode/json string->instant
                         :json-schema {:type "string", :format "date-time"}
                         :gen/gen (gen/fmap #(.toInstant ^Date %) (mg/generator inst?))}})))

(m/form [:map [:x instant]])
; => [:map [:x instant]]

(m/validate instant (Instant/now))
;=> true

(-> [:map [:x instant]]
    (m/explain {:x "kikka"})
    (me/humanize))
; => {:x ["should be instant"]}

(json-schema/transform [:map [:x instant]])
;{:type "object"
; :properties {:x {:type "string"
;                  :format "date-time"}}
; :required [:x]}

(m/decode
  [:map [:x instant]]
  {:x "2021-02-09T13:49:44.419Z"}
  (mt/json-transformer))
; => {:x #object[java.time.Instant 0x5aed36d3 "2021-02-09T13:49:44.419Z"]}

(mg/generate [:map [:x instant]])
; => {:x #object[java.time.Instant 0x4878fa62 "1970-01-01T00:00:00.295Z"]}
#2021-02-0913:54ikitommie.g. m/-simple-typeallows to (easily?) build custom schemas that cover all the aspect: transforming, humanized errors, generators, json-schema, etc.#2021-02-0913:55ikitommimuch of the core itself is built on top of that.#2021-02-0913:56ikitommi:type-properties allow one to hide implementation details. one could do the same fully using normal schema properties, but all the details would be in your face when you look at the schema form.#2021-02-0914:18danielnealthanks @ikitommi, that looks great#2021-02-1215:16kwrooijenHi, is this a bug?
(def model
  [:map
   [:my/field {:default nil} :int]])

(m/decode model params mt/default-value-transformer)
;; => {}
I'm expecting {:my/field nil} here
#2021-02-1215:21ikitommi@kevin.van.rooijen a bug, PR welcome#2021-02-1215:21kwrooijenWill do 👍#2021-02-1215:21kwrooijenKind of have dejavu with this bug#2021-02-1215:21kwrooijenhttps://github.com/metosin/malli/pull/223/files lol 😅#2021-02-1215:24ikitommifalsey is hard.#2021-02-1215:24kwrooijenGood thing Clojure only has 2 falsey values.. instead of some other languages I won't name#2021-02-1422:29caumondHi guys, I have tried to draw malli's dot representation. Everything's fine until I try to print a [:<= 20]or [:< 20] . In both case, it seems to be an invalid sequence. At least tangle doesn't know how to handle it. Everything's fine if I replace [:<= 20 in the string built by transform.#2021-02-1422:29caumondIs it a PR?#2021-02-1503:23aaron51Is there an easy way to use Malli in clojure.test/is? I’d like to assert that m/validate returns true, and show m/explain output when it fails.#2021-02-1504:27aaron51(is (validate schema data) (humanize (explain schema data))) — anything like this built in? Or just validate! that throws on failure?#2021-02-1507:55borkdudeWrite a function?#2021-02-1508:40ikitommithere are no test helpers atm, but should be easy to do in user space as borkdude suggests, Are there good helpers / utilities for this (testing) in spec? in schema?#2021-02-1522:38aaron51Yes, it is straightforward enough to write it in user space. But I thought it would be a common case… Found a few similar things: • schema: https://github.com/plumatic/schema/blob/master/test/clj/schema/test_macros.clj • schema: https://github.com/plumatic/schema/blob/master/test/cljx/schema/core_test.cljx • clojure-expectations has “spec expectations” https://github.com/clojure-expectations/clojure-test#2021-02-1522:40borkdudeI also have this one: https://github.com/borkdude/respeced This was written to test fdef specs.#2021-02-1620:04ikitommiwrote an issue out of this: https://github.com/metosin/malli/issues/369.#2021-02-1508:34ikitommi@caumond PR welcome#2021-02-1508:42ikitommiwip: function generators generate correctly function with different arities
(def f
  (mg/generate
    [:function
     [:=> :cat int?]
     [:=> [:cat int?] nat-int?]]))

(f)
;=> 132816
;=> -7823115
;=> -36
;=> -97
;=> 13412759
;=> 1444

(f 1)
;=> 1038018
;=> 11009747
;=> 8
;=> 59186626
;=> 10
;=> 5373734

(f "1")
; =throws=> :malli.generator/invalid-input {:schema [:cat int?], :args ["1"]}

(f 1 2)
; =throws=> :malli.generator/invalid-arity {:arity 2, :arities #{0 1}, :args (1 2), :schema [:function [:=> :cat int?] [:=> [:cat int?] nat-int?]]}
#2021-02-1508:44borkdudeis this in a branch somewhere?#2021-02-1508:45ikitomminot yet, needs a new -min-count protocol method to regex-schma to resolve the arities from :=> schemas. few hours away from a branch.#2021-02-1508:49borkdudecool. how are you generating the fn per arity?#2021-02-1508:53ikitommipushed the current: https://github.com/metosin/malli/blob/fn_new/src/malli/generator.cljc#L104-L129#2021-02-1508:54ikitommibut that code doesn’t infer the arity, needs to set manually:
(def f
  (mg/generate
    [:function
     [0 [:=> :cat int?]]
     [1 [:=> [:cat int?] nat-int?]]]))
#2021-02-1508:59borkdude:thumbsup:#2021-02-1819:56Daniel MiladinovIs it possible to use a `[:map …]` spec to describe interactions / interdependencies with map keys / values? Like, “either this key or that key must be set”, or “these two keys must either both have empty values or both be set”#2021-02-1819:58borkdudeThis is probably done using a predicate on the map?#2021-02-1819:58borkdude(Guess)#2021-02-1819:59Daniel MiladinovI mean, I know I can write a [:fn …] spec and do whatever I need with the map validation, but I was wondering if there was anything more declarative.#2021-02-1820:49ikitommi@daniel.miladinov if you can cook up a good declarative syntax for defining the rules, I can try to summon a custom schema to do that. Declarative rule systems are not always simple.#2021-02-1820:50ikitommiNot sure if something like meander could be used to declare rules like that. Bridging Malli and Meander might be fun#2021-02-1914:24dangercoderHi! According to the docs there is a parse function which im very much interested in.
(m/parse
  [:* [:cat*
       [:prop string?]
       [:val [:alt*
              [:s string?]
              [:b boolean?]]]]]
  ["-server" "foo" "-verbose" true "-user" "joe"])
In what namespace does this function live? I tried looking through the code and could not find it. Also tried the example (I thought, well, maybe m is bound to malli.core) 🙂
#2021-02-1914:26ikitommiit’s in master, and it’s malli.core/parse.#2021-02-1914:26ikitommirelease soon, just need more tests and some docs.#2021-02-1914:27dangercoderI see its me looking in the wrong branch, sorry. I looked in 0.2.1#2021-02-2115:26ikitommiadded more info on function schema explain:
(def MyFunctionSchema
  (m/schema
    [:function
     [:=> [:cat :int] :int]
     [:=> [:cat :int :int [:* :int]] :int]]
    {::m/function-checker mg/function-checker}))

(m/explain
  MyFunctionSchema
  (fn
    ([x] x)
    ([x y & zs] (apply max x y zs))))
; => nil

(m/explain
  MyFunctionSchema
  (fn
    ([x] x)
    ([x y & zs] (str (apply max x y zs)))))
;{:schema [:function 
;          [:=> [:cat :int] :int] 
;          [:=> [:cat :int :int [:* :int]] :int]],
; :value #object[],
; :errors (#Error{:path [],
;                 :in [],
;                 :schema [:function 
;                          [:=> [:cat :int] :int] 
;                          [:=> [:cat :int :int [:* :int]] :int]],
;                 :value #object[],
;                 :check ({:total-nodes-visited 2,
;                          :depth 1,
;                          :pass? false,
;                          :result false,
;                          :result-data nil,
;                          :time-shrinking-ms 0,
;                          :smallest [(0 0)],
;                          :malli.generator/explain-output {:schema :int,
;                                                           :value "0",
;                                                           :errors (#Error{:path []
;                                                                           :in []
;                                                                           :schema :int
;                                                                           :value "0"})}})})}
#2021-02-2115:29ikitommireally happy how good the function schemas are now, but still - it’s just elegant (and slow) way to verify things at runtime, e.g. not a type system.#2021-02-2115:32ikitommiin master#2021-02-2115:33borkdudecompared to clojure.spec, how slow?#2021-02-2115:44ikitommithe sequence schemas are at least an order of magnitude faster than spec in initial tests. Full function verification uses test.check, I think that dominates, and quess they are about the same.#2021-02-2115:47ikitommibut, having arity info available for fns at runtime (in clojure.core) would help to see that fns are correct on arity, without testing it and getting arityexceptions - costs a lot.#2021-02-2116:13borkdudeMaybe you can combine that with static analysis (e.g. from clj-kondo)?#2021-02-2118:09ikitommiI guess it could be done using a clj-kondo hook? with defaults, the test.check runs 19ms for a function on my machine. that’s 500 funcs per sec (on 1 core?). errors are faster, the given example fails fast on invalid return, under 1ms.#2021-02-2120:03borkdudeI mean: > but, having arity info available for fns at runtime (in clojure.core) would help to see that fns are correct on arity, without testing it and getting arityexceptions - costs a lot. You could use clj-kondo to find the correct arities perhaps#2021-02-2118:10ikitommii believe it could be made faster, less iterations on arguments etc.#2021-02-2118:11ikitommi[metosin/malli "0.3.0-SNAPSHOT"] has the stuff in.#2021-02-2119:59ikitommiwith 10 iterations, it drops to 0.3ms per fn.'#2021-02-2120:42emccue> e.g. not a type system.#2021-02-2120:42emccuei mean#2021-02-2120:43emccuefor me, it would serve much the same purpose i'd think#2021-02-2120:43emccueor rather - the same workflow#2021-02-2120:43emccuemaybe i'm missing a worldview though#2021-02-2121:07borkdudeBtw, this sounds interesting, but I haven't heard anything about it yet: https://www.fulcrologic.com/copilot#2021-02-2208:28ikitommilooking forward for that. I’m watching the guardrails repo, but think that’s going to be much more.#2021-02-2208:28ikitommi> (extensible, but starting with Clojure spec)#2021-02-2208:54borkdudeYeah, but will be a commercial product, so hard to contribute and see what's going on probably#2021-02-2211:06dharriganLooking forward to the new function schema - just saw the tweet!#2021-02-2211:53ikitommi@emccue Malli is a dynamic schema/type system, it’s not a static type system. tools like clj-kondo and typed clojure can do static analysis, maybe copilot too? Would love to see tooling get better and happy to help, but no time or skills to do anything non-trivial except to integrate into existing tooling. My point was that resolving fn arity using generative testing or 3rd party tools (clj-kondo) is a hard way to do something that would be easy to do in the core language itself. With core clojure: does a ring middleware chain would work for an async (3) arity? run and see if it throws arityexception 😞#2021-02-2211:55ikitommiI’m hope and think we’ll see great new developer tooling for clojure in 2021, from the community.#2021-02-2214:44ikitommiadded more benchmarks to repo, parsing vs spec:
;; 44µs
(let [spec (s/* (s/cat :prop string?,
                       :val (s/alt :s string?
                                   :b boolean?)))
      parse (partial s/conform spec)]
  (cc/quick-bench
    (parse ["-server" "foo" "-verbose" "-verbose" "-user" "joe"])))

;; 2.5µs
(let [schema [:* [:cat*
                  [:prop string?]
                  [:val [:alt*
                         [:s string?]
                         [:b boolean?]]]]]
      parse (m/parser schema)]
  (cc/quick-bench
    (parse ["-server" "foo" "-verbose" "-verbose" "-user" "joe"])))
#2021-02-2216:23Jakub ZikaHi guys, I am trying to generate some random date with (mg/generate inst?) but i am not sure how :seed or :size works here. I am getting 1969 or 1970 in 99% cases. How to get a date like any >1980? Thanka dwh-replicator.database> (mg/generate inst? {:seed 20}) ;; => #inst "1970-01-01T00:00:00.000-00:00" dwh-replicator.database> (mg/generate inst? {:seed 42}) ;; => #inst "1970-01-01T00:00:00.000-00:00" #2021-02-2216:43juhoteperiI think in this case Malli just use spec.alpha generator for instants: https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L178 https://github.com/clojure/spec.alpha/blob/31165fec69ff86129a1ada8b3f50864922dfc88a/src/main/clojure/clojure/spec/gen/alpha.clj#L160#2021-02-2216:55pithyless@jakub.zika-extern there's a good talk that explains how :seed and :size interact when writing custom generators. It even mentions the datetime problem @ 31:54. https://youtu.be/F4VZPxLZUdA?t=1911 1. The proposed solution in the talk is a custom generator that splits the datetime into separate domain bits (year / month / day / etc.) 2. A different solution (as seen e.g. https://github.com/nasa/Common-Metadata-Repository/blob/master/common-lib/src/cmr/common/test/test_check_ext.clj#L255-L257) is to change the way you choose an initial seed integer (that is used to coerce to date).#2021-02-2216:56Jakub ZikaThank you!#2021-02-2221:14Jakub ZikaI am mapping DB types to clojure types, so varchar(255) goes to [string? {:min 0 :max 255}] etc. I am using these derived schemas to: 1. validate data : (malli/validate [string?] "hoy") 2. generate sample data : (malli.generator/generate [string?]) When I will go for custom generators then I will have to create new validators, correct?#2021-02-2221:19pithylessYou'll need to pass in the custom generator to the spec (not necessarily when validating, but at the very least wen generating sample data).#2021-02-2221:20pithylessso instead of
[string? {:min 0 :max 255}]
you'll have to wrap it, eg:
[:and {:gen/elements ["kikka" "kukka" "kakka"]} [string? {:min 0 :max 255}]]
#2021-02-2221:20pithylesswhere you use one of :gen/elements, :gen/gen, :gen/fmap, etc.#2021-02-2221:25pithyless^ I think you might even be able to just change the default :gen/gen of the specific type by modifying the malli registry globally (but then you're changing it for everything... which has its own issues ;))#2021-02-2221:26pithylessOR, perhaps make a custom registry where you override the :gen/gen for the basic types you're interested in (e.g. instant); and then pass in the custom registry only when generating sample data#2021-02-2306:26ikitommino need to wrap into :and, this works too: [:string {:min 0, :max 100, :gen/min 10, :gen/max 20, :gen/fmap (partial * 2)}]#2021-02-2306:27ikitommithe default schema generators are implemented as multimethods, so for global effects, one can just re-mount generator fof 'inst? for example. not recommended.#2021-02-2218:07Alex WhittWould anyone like to jump on this discussion thread I created on the subreddit? (I'd prefer to keep it there so it doesn't disappear behind Slack's paywall) https://www.reddit.com/r/Clojure/comments/lpv8ok/spec_vs_malli/#2021-02-2318:12ikitommifinal change to figure out proper names for named branch -variants for: :cat, :alt and :or. they are currently :cat*, :alt* and :or* :face_with_rolling_eyes:
(m/parse
  [:*
   [:alt* 
    [:s :string]
    [:i :int]]]
  [1 "2" 3 "4"])
; => [[:i 1] [:s "2"] [:i 3] [:s "4]]
#2021-02-2318:14ikitommi:catn, :altn & :orn?#2021-02-2318:14ikitommi:cat-named, :alt-named & :or-named?#2021-02-2318:15ikitommi:alley-cat, :danger-mouse & :skeletor?#2021-02-2407:28ordnungswidrig:cat-tag :alt-tag :or-tag ?#2021-02-2413:05Vincent Cantini suggest :named-cat, :named-alt, :named-or#2021-02-2413:23danielneal[:cat {:named? true}]?#2021-02-2814:09ikitommiusing properties would be nice for users, but would make the parser more complex. and updating properties might change the behaviour of the schema silently:
[:cat {:named? true} [:tuple int?]]
#2021-02-2814:09ikitommi
[:cat {:named? false} [:tuple int?]]
… is a totally diferent thing.
#2021-02-2507:17steveb8nQ: can I use Malli inside a babashka script? (i.e. does it run inside sci?)#2021-02-2507:17steveb8nthis would be useful for validating tools.cli parsed options#2021-02-2507:43borkdudeNot yet, but feel free to post an issue, so we can gather community feedback. Meanwhile there are two options: spartan.spec and minimallist both work #2021-02-2508:32ikitommithere is https://github.com/metosin/malli/issues/302. Help most welcome#2021-02-2508:35borkdudeI think it will be quite a lot of work to port malli to bb compatible code. @ikitommi What is the API stability of malli? At one point we could add it to bb proper perhaps. I want to consider this, but also want some feedback from the "bb community" on this#2021-02-2508:37ikitommiVery stable. the named-schemas will change, but won’t change after release.#2021-02-2508:37ikitommipublic api has been immutable for 6+ months, no plans on breaking.#2021-02-2508:39steveb8nthanks guys. since it’s only tools.cli in CI (i.e. almost never changes), I can write a custom vaildator instead#2021-02-2508:40borkdude@U0510KXTU OK. This lib https://github.com/green-coder/minimallist also works with bb btw#2021-02-2508:42steveb8ncool. I’ll give it a try. thx#2021-02-2508:44steveb8nI like that it explicitely does not try to be fast. probably means they can deliver features faster#2021-02-2508:44steveb8nunlike Malli where I’m getting value out of the perf vs spec. perf is a feature#2021-02-2508:45borkdudeperf is a feature and the style of writing code (reifying Java classes) isn't well supported by bb from source for any Java class :)#2021-02-2508:46borkdudeWell, it is supported for records, etc, but definitely not fast#2021-02-2508:46ikitommimalli reifies just protocols, does that work ok?#2021-02-2508:46ikitommiis there something that we could do on malli side to make it work?#2021-02-2508:47steveb8nit’s all tradeoffs. I’m glad Tommi likes fast things 🙂#2021-02-2508:47borkdudeit does work:
$ bb -e '(defprotocol Foo) (instance? Foo (reify Foo))'
true
#2021-02-2508:48borkdude@ikitommi Maybe using reader conditionals helps, #?(:bb :foo :clj :bar)#2021-02-2508:48borkdudefor the parts that bb doesn't support#2021-02-2508:48ikitommisure, happy to add. which parts? 🙂#2021-02-2508:48borkdudetry and you will find out :) happy to comment on those parts#2021-02-2508:49ikitommithe protocol cache thing at least? (hacking over dead slow satisfies?)#2021-02-2508:49ikitommijust write a bb script of some malli code and see what breaks?#2021-02-2508:50borkdudeyes. in the malli repo, write a script like:
(require '[babashka.classpath :as cp])
(cp/add-classpath "src")
and then
(require '[malli.core :as m])
#2021-02-2508:50ikitommiexample/test-bed most welcome :)#2021-02-2508:51ikitommithat simple. cool.#2021-02-2508:51ikitommiwill add that to our next tech/hack-friday, which is… tomorrow.#2021-02-2508:53ikitommiso, does the :bb conditional work already?#2021-02-2508:53borkdude@ikitommi Actually, this is better:
(ns bb-script)

(require '[babashka.deps :as deps]
         '[clojure.edn :as edn])

(deps/add-deps (edn/read-string (slurp "deps.edn")))

(require '[malli.core :as m])
#2021-02-2508:53borkdude@ikitommi Yes, :bb is prioritized over :clj#2021-02-2508:53borkdudeit's order dependent#2021-02-2508:56borkdude@ikitommi So the first hack I did:
(ns malli.sci
  #
#2021-02-2508:56borkdudeThe nest error:
----- Error --------------------------------------------------------------------
Type:     java.lang.Exception
Message:  Unable to resolve classname: java.util.ArrayDeque
Location: /private/tmp/malli/src/malli/impl/regex.cljc:35:3

----- Context ------------------------------------------------------------------
31:   recognition for `validate`."
32:
33:   (:refer-clojure :exclude [+ * repeat cat])
34:   (:require [malli.impl.util :as miu])
35:   #?(:clj (:import [java.util ArrayDeque])))
      ^--- Unable to resolve classname: java.util.ArrayDeque
#2021-02-2508:56borkdudeSo we don't have that class in bb (yet)#2021-02-2508:57borkdudeI see that for cljs you didn't use it#2021-02-2508:58borkdudeI guess for bb you can take the :cljs branches there#2021-02-2508:59borkdudeThen after doing that, I run into:
(deftype ^:private CacheEntry [^long hash f ^long pos regs])
#2021-02-2509:00borkdudeThere are other deftypes in regex.cljc#2021-02-2509:00ikitommiwhat would be a good workaround for those?#2021-02-2509:00ikitommior, will bb support those at some point?#2021-02-2509:01ikitommigood to understand how to create bb-compatible code for the future.#2021-02-2509:01borkdudeSo these are the types of things that are not supported yet. Classes can be added, but deftype isn't supported. I don't know what to do with deftype yet. Records are "faked" using normal maps + metadata.#2021-02-2509:04borkdudeI have seen a similar thing with meander vs matchete. Matchete is written in a "simple" Clojure style so it just works with bb out of the box. https://github.com/xapix-io/matchete Where meander focuses on performance and this results in incompatible code.#2021-02-2509:05borkdudeMeander now has an "interpreter" which is compatible with bb which also more flexible than the macro style enforced in the main lib#2021-02-2512:13borkdudeIs there a way to get the humanized string for a schema, without any input?
(prn (me/humanize [:< 100]))
;;=> "should be smaller than 100"
#2021-02-2514:26ikitommioh, with nil value you mean?#2021-02-2514:32borkdudewith no value at all#2021-02-2514:32borkdudeit seems that the returned string doesn't depend on the input#2021-02-2514:19ikitommiyou can either:
[:int {:max 100}]
or:
[:< {:error/message "should be smaller than 100"} 100]
#2021-02-2514:20ikitommithere are set of "type" schemas, which read properties: :int, :string , ...#2021-02-2514:22ikitommimight map all predicates schemas into these internally, which would make things like JSON Schema transformations & normal transformations simpler, need just to be defined for the base type, not to all predicates.#2021-02-2514:22ikitommipos-int? -> [:int {:min 1}]...#2021-02-2514:53borkdude@ikitommi What I mean is: you can get a string from humanize but this needs some input too which you first have to validate. However, the resulting string doesn't seem to be related to the input at all. This made me wonder: is it possible to get some "model" error message from a schema only, without validating anything#2021-02-2514:53borkdudeThis can then be passed to the second arg of the :validate tuple in tools.cli#2021-02-2514:54borkdudeNot that important, I can make the string itself manually or by validating some dummy input#2021-02-2516:09ikitommimalli supports both fixed error messages and functions generating error messages, the two examples here: https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L62-L68#2021-02-2516:10ikitommi... and messages can be localized. So, no simple way to do without value.#2021-02-2516:11ikitommithere are helpers in malli.error to pull out the error message/fn out of a schema, but value is needed for the fn case#2021-02-2516:14ikitommialso, one schema can emit many different errors, humanized by the error type. Maps emit extra-keys, missing-keys, missspelled-keys, etc.#2021-02-2610:49borkdudeDid this experiment. It brings malli to babashka via a pod. Due to pod limitations it might not bring you all the features, but might still be useful somehow. Feedback welcome 😊#2021-02-2619:57bartukanot sure if ask in #malli or #reitit channel, but nesting routes with path params are not merged correctly. I found this issue https://github.com/metosin/reitit/issues/462 there is any update on this?#2021-02-2620:05juhoteperiNo updates yet. It is Reitit issue, Malli has malli.util/merge function that would work here, but Reitit doesn't have option to control how the values are merged, and because it is currently done using external library (https://github.com/weavejester/meta-merge) it is not simple to change. Might have to copy meta-merge logic to Reitit and extend with controls to use custom function in certain paths.#2021-02-2620:56juhoteperihttps://github.com/metosin/reitit/pull/474 adds tests and found the place where we could call malli merge#2021-02-2620:05juhoteperiNo updates yet. It is Reitit issue, Malli has malli.util/merge function that would work here, but Reitit doesn't have option to control how the values are merged, and because it is currently done using external library (https://github.com/weavejester/meta-merge) it is not simple to change. Might have to copy meta-merge logic to Reitit and extend with controls to use custom function in certain paths.#2021-02-2620:56juhoteperihttps://github.com/metosin/reitit/pull/474 adds tests and found the place where we could call malli merge#2021-02-2620:58ikitommithere is also the nil-issue, that won't be fixed in meta-merge. Would simplify a lot how reitit route data can be used. metosin/ctrl-merge, right?#2021-02-2620:58ikitommihttps://github.com/weavejester/meta-merge/issues/11#2021-02-2814:06ikitomminaming is hard. to merge or not to merge: https://github.com/metosin/malli/pull/378#2021-03-0116:50ElsoDidn't really know where to put it, more a general metosin issue, but I'm getting Syntax error macroexpanding at (core.clj:152:3). Execution error (ClassNotFoundException) at jdk.internal.loader.BuiltinClassLoader/loadClass (BuiltinClassLoader.java:581). com.fasterxml.jackson.core.util.JacksonFeature when loading metosin/jsonista 0.3.1#2021-03-0120:22mynomotoI'm trying the current master a58e04b265b1b6658bb9fe791fd103cfab452e53 for clj-kondo linting of function specs and it looks like that the configuration is generating the :ret key as shown on the readme. Was that removed on purpose, or am I doing something wrong?#2021-03-0120:27mynomotoI think https://github.com/metosin/malli/commit/0a22ccb1036c25b12f4b8c0d0f1dc20682aa6ca1 broke it, will create a pr.#2021-03-0201:41bartukaI am trying to use :* and got this message {:type :malli.core/invalid-schema, :data {:schema :*}} there is some setup to enable regex-like support?#2021-03-0201:51bartukaI think https://github.com/metosin/malli/blob/a58e04b265b1b6658bb9fe791fd103cfab452e53/src/malli/core.cljc#L1873 (sequence-schemas) happened after the last release hehe#2021-03-0202:30mynomotoIf you are using tools-deps you can reference a github commit.#2021-03-0208:12Yevgeni TsodikovHi all, I’m trying to define a map with several optional fields, that at least 1 of them must exist. For example, :a and :b are optional, :c is required.
{:a 1 :c 1}      ; valid
{:b 1 :c 1}      ; valid
{:a 1 :b 1 :c 1} ; valid
{:c 1}           ; nope, invalid
As a reference, with `spec` , I’d write something like
(s/def ::my-map
  (s/keys :req-un [::c (or ::a ::b)]))
#2021-03-0209:10ikitommi@evg.tso you can compose with :and, see example here: https://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&amp;schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D#2021-03-0209:39Yevgeni TsodikovThe thing with :and is that is doesn’t survive mu/merge very well. To be more specific, my use case is with reitit. There are several routes with common fields, but each route has specific fields that are different from one another. • POST /share/this would accept either :a or :b and must contain :c (for example {:a 1 :c 1}POST /share/that would accept either :a or :b and must contain :d (for example {:a 1 :d 1}
.
└── schema with optional fields (at least one must exist)
    ├── mu/merge-ed schema with specific fields
    └── mu/merge-ed schema with specific fields
#2021-03-0209:41Yevgeni Tsodikov
(mu/merge 
  [:map [:specific-field string?]]
  [:and
   [:map [:x int?] [:y int?]]])
=> [:and [:map [:x int?] [:y int?]]]
#2021-03-0209:44ikitommiI see. What if :and was considered as a constrained-kinda thing, where the first value is the actual schema and the rest are just extra rules for that. Would solve a lot of things. :thinking_face:#2021-03-0209:45ikitommineed to check would it break existing contracts or not.#2021-03-0209:47ikitommiwould this be more correct:
(mu/merge 
  [:map [:specific-field string?]]
  [:and
   [:map [:x int?] [:y int?]]
   map?]])
=> [:and [:map [:spesific-field string?] [:x int?] [:y int?]] map?]
#2021-03-0209:47Yevgeni TsodikovThat would be great!#2021-03-0209:50Yevgeni TsodikovIn most of my schemas I use :and to add additional data like a :fn or a custom error message.#2021-03-0212:06ikitommi[metosin/malli "0.3.0"] 🥳 🥳 🥳#2021-03-0212:40delaguardoHi! I’m using malli together with reitit and reitit-swagger and I found a problem when I start using custom malli’s registry. tldr; generated swagger partials for an endpoint contains “definitions” key which is not valid according to swagger2 spec. also it messes up with references trying to point to one of defined schemas. As a workaround I added a middleware to walk throw generated object which will be encoded as json and pull all definitions to the top level but it looks ugly and not generic enough. Also there is a small inconsistency in encoding keywords to json. https://malli.io/?value=%5B1%20%5B2%20%5B3%20%5B4%20nil%5D%5D%5D%5D&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%3Afoo%2FConsCell%20%5B%3Amaybe%20%5B%3Atuple%20%3Aint%20%5B%3Aref%20%3Afoo%2FConsCell%5D%5D%5D%7D%7D%0A%20%3Afoo%2FConsCell%5D here is an illustration — :$ref key contains a string "#/definitions/:foo/ConsCell" with : inside. and there is a key :foo/ConsCell in :definitions after encoding to json this schema will be invalid because of : present in “$ref” key#2021-03-0213:02ikitommiI see. PR would be welcome to fix this. Maybe stringify all reference keys?#2021-03-0213:05delaguardoI would like to try ) where such fix should go? reitit-swagger or malli itself?#2021-03-0213:14ikitommiIn malli, either malli.json-schema or malli.swagger ns.#2021-03-0213:14delaguardocool, will be back with PR )#2021-03-0214:08stathissideriscongrats on the release!#2021-03-0413:24borkdudeNice tweet @mikethompson :) https://twitter.com/wazound/status/1367303612028751872#2021-03-0413:25borkdudeI hear similar sentiments in the latest #defnpodcast - there they called it the Osborne effect. https://en.wikipedia.org/wiki/Osborne_effect#:~:text=The%20Osborne%20effect%20is%20a,announcing%20a%20future%20product%20prematurely.#2021-03-0413:30borkdudeI have the exact same problem with babashka: waiting for spec2 so not including spec1: and this can result in years of having nothing perhaps#2021-03-0413:31mikethompsonI didn't hear that podcast, but yeah, I feel like Clojure needs an outcome. We're stuck in something of a twilight zone. And this is an important issue. And right now Malli is just better. Unless there is something pending with spec ... it feels like we need permission to get on with making Malli "it"#2021-03-0413:33borkdudeIn some way, no matter how good your library is, you can't compete with core since people will just believe that "core is better" no matter what you do, which isn't fair maybe. But "better" is probably not the right way to describe it: spec and malli have different approaches. spec is more of an RDF approach where each unique attribute name describes a schema#2021-03-0413:34mikethompsonI'm the same as you: should re-frame support spec or Malli. I've held off for a long time.#2021-03-0413:35borkdudeWhy should re-frame have to choose here? Can't this be decided in some add-on lib? Why do you need to address this concern in re-frame itself?#2021-03-0413:35mikethompsonI could#2021-03-0413:35borkdudeRe-frame events are already associated with globally namespaced keywords which maybe aligns well with spec#2021-03-0413:36mikethompsonBut if ever spec is ratified as the one true way for Clojure, I'd like for it to be the one true way for re-frame too#2021-03-0413:36mikethompsonAnyways#2021-03-0413:37borkdudeMaybe it can be made pluggable#2021-03-0413:45juhoteperiReitit supports all three so the application can choose.#2021-03-0413:45juhoteperiPluggable solution is probably also quite important in Cljs apps, because either Spec or Malli both add about 100KB JS to the output bundle (before gzip).#2021-03-0413:46juhoteperiI was surprised by this recently when I checked project output report in app that uses Malli, but re-chain uses Spec internally: https://github.com/ingesolvoll/re-chain/issues/6#2021-03-0415:38ikitommiInteresting discussion. About Malli cljs-size. The core has been developed DCE in mind, you can make a Malli bundle with just validation of strings, numbers, maps, vector and sets and it’s few kilos zipped. Currently, many extensions (humanized errors, generation, json schema etc) are implemented using multimethods, so pulling anything out of those, makes it much bigger.#2021-03-0415:39ikitommiThe Code Size Expression Problem.#2021-03-0415:40ikitommi#2021-03-0415:40ikitommihere’s the bare-bones malli.#2021-03-0415:42ikitommi@mikethompson what would the spec/malli integration look like? do you need something?#2021-03-0416:02juhoteperiAnd Reitit Malli coercion is built for Ring model, so it includes lots of features that are unnecessary for frontend routing. When we get around to implementing separate frontend coercion, it will drop dependency on some unused parts.#2021-03-0416:03juhoteperiLike the json schema generation parts.#2021-03-0416:12emccue^i'm not opposed to the json schema stuff but it does feel odd that it isn't a separate artifact#2021-03-0416:15juhoteperiIt is separate namespace on Malli. On Reitit coercion impl. the parameter validation and generating the json-schemas for routes is closely related so they are on the same module currently.#2021-03-0416:17ikitommiyes, there should be a lite-version of reitit coercion, with just the encoding & decoding part - without throwing exceptions. Frontend would use that: simple, pure and small.#2021-03-0416:18ikitommimalli is designed so that malli.core is the essential ns, all others are optional (ok, there is nowadays malli.impl too). It has been easier to optimize the whole as everyhing is in single repo.#2021-03-0417:39arundilipanHi everyone, I’m not sure if this is a bug, but I’m running into a quirk with the validation and error reporting of :catn schemas#2021-03-0417:40arundilipanFor example if you had a schema like:
[:catn 
  [:amount [:fn {:error/fn '(fn [{:keys [value]} _] (str "Received value " value ", should be > 0"))
        :decode/string malli.transform/-string->double}
    '#(> % 0)]]
  [:type [:enum "A" "B" "C"]]]
#2021-03-0417:45arundilipanIf I try and (malli.core/explain schema [1.0 "D"]) , and humanize it, the error I get back is "Received value 1.0, should be > 0" , rather than "should be one of "A", "B", or "C"#2021-03-0417:47borkdudeIsn't the idea of :catn that you provide names for the schema elements?#2021-03-0417:47borkdudeElse you should just use :cat#2021-03-0417:56arundilipanFixed the example, I do intend to use :catn to provide names for the schema elements#2021-03-0420:00ikitommi@arundilipan seems to work:
(-> [:catn
     [:amount [:fn {:error/fn '(fn [{:keys [value]} _] (str "Received value " value ", should be > 0"))
                    :decode/string malli.transform/-string->double}
               '#(> % 0)]]
     [:type [:enum "A" "B" "C"]]]
    (m/explain [1.0 "D"])
    (malli.error/humanize))
; => [nil ["should be either A, B or C"]]
#2021-03-0420:11borkdudeshould :catn not give back error messages by name?#2021-03-0420:18ikitommierror messages by name?#2021-03-0420:19borkdude{:amount nil :type ["should be ..."]}#2021-03-0420:19arundilipanThat’s for :map i think#2021-03-0420:19borkdudeare the positions in :cat always unambiguous? I know for spec they are certainly not with s/cat#2021-03-0420:19ikitommihmm. not sure. the current humanize just mimics the raw result, not parsed one.#2021-03-0420:20ikitommibut the info is there:#2021-03-0420:20ikitommi
(-> [:catn
     [:amount [:double {:min 0}]]
     [:type [:enum "A" "B" "C"]]]
    (m/explain [1.0 "D"]))
;{:schema [:catn [:amount [:double {:min 0}]] [:type [:enum "A" "B" "C"]]],
; :value [1.0 "D"],
; :errors (#Error{:path [:type 0], :in [1], :schema [:enum "A" "B" "C"], :value "D"})}
#2021-03-0420:21ikitommie.g. it’s a sequence, errors are positioned by :in.#2021-03-0420:21borkdudemaybe for tools like expound it would be nice to have the humanized error along with the path?#2021-03-0420:21borkdudeah I see#2021-03-0420:21borkdudeso using the explain output you can get to the error message by path#2021-03-0420:21borkdudeusing get-in or so#2021-03-0420:23ikitommifor map, the explain info is:
(-> [:map
     [:amount [:double {:min 0}]]
     [:type [:enum "A" "B" "C"]]]
    (m/explain {:amount 1.0
                :type "D"}))
;{:schema [:map [:amount [:double {:min 0}]] [:type [:enum "A" "B" "C"]]],
; :value {:amount 1.0, :type "D"},
; :errors (#Error{:path [:type 0], :in [:type], :schema [:enum "A" "B" "C"], :value "D"})}
#2021-03-0420:24ikitommibut yes, the branch name is available in the error data for :catn, to be printed nicely etc.#2021-03-0420:26ikitommi@arundilipan merged a humanized error for :double, so this works now too:
(-> [:catn
     [:amount [:double {:min 0}]]
     [:type [:enum "A" "B" "C"]]]
    (m/explain [1.0 "D"])
    (malli.error/humanize))
; => => [nil ["should be either A, B or C"]] 
#2021-03-0421:10borkdude
user=> (require '[malli.error :as e])
nil
user=> (require '[malli.core :as m])
nil
user=> (e/humanize (m/explain [:cat int? int? [:? int?] [:? string?]] [1 2 :foo]))
[nil nil ["should be an int" "should be a string" "unknown error"]]
Looks legit, except maybe for the "unknown error"?
#2021-03-0421:13borkdudeThat's probably better expressed as:
(m/explain [:cat int? int? [:orn [:x int?] [:y string?]]] [1 2 :foo])
Just wanted to see what malli would make of it
#2021-03-0422:04ikitommigood catch @borkdude, the regex error types didn’t have humanized forms. fixed in master:
(-> [:cat int? int?]
    (m/explain [1])
    (me/humanize))
; => [nil ["end of input"]]

(-> [:cat int? int?]
    (m/explain [1 2 3])
    (me/humanize))
; => [nil nil ["input remaining"]]

(-> [:cat int? int? [:? int?] [:? string?]]
    (m/explain [1 2 :foo])
    (me/humanize))
; => [nil nil ["should be an int" "should be a string" "input remaining"]]
#2021-03-0422:05borkdudenice!#2021-03-0422:08ikitommiwith :or:
(-> [:cat int? int? [:or int? string?]]
    (m/explain [1 2 :foo])
    (me/humanize))
; => [nil nil ["should be an int" "should be a string"]]
#2021-03-0422:14borkdudeyeah, that already worked right?#2021-03-0504:57bedersquick question with regards to properties. I’m trying to put some properties into my schemas, but can’t seem to be getting them back. What am I doing wrong here?
; malli 0.3.0:
; [malli.registry :as registry]
(def registry
  (registry/composite-registry
    malli/default-registry
    {:common/single-line [:re {:bubu :lala} #"^[^\r\n]*$"]}))

(malli/properties :common/single-line {:registry registry})
=> nil
Expected: {:bubu :lala}
#2021-03-0507:34ikitommi@beders oh, that’s not good. registry interally wraps the registered schema instances into :malli.core/schema , which is an eager reference type. When you pull out an instance schema from a registry, you get the reference back. it mostly a pass-through, e.g. calling -validatorto the reference return the validator of the referenced schema. But: for some reason, the current impl returns the reference properties and options if asked. I think it’s a bad feature, should be changed.#2021-03-0507:36ikitommineed to think how that effects other things. before that, you m/deref safely:#2021-03-0507:37ikitommi
(def registry
  (mr/composite-registry
    m/default-registry
    {:common/single-line [:re {:bubu :lala} #"^[^\r\n]*$"]}))

(-> (m/schema :common/single-line {:registry registry})
    (doto prn)
    (m/deref)
    (doto prn)
    (m/properties))
;:common/single-line
;[:re {:bubu :lala} #"^[^\r\n]*$"]
;=> {:bubu :lala}
#2021-03-0507:38ikitommiworkaround for now:
(def registry
  (mr/composite-registry
    m/default-registry
    {:common/single-line [:re {:bubu :lala} #"^[^\r\n]*$"]}))

(defn schema [?schema]
  (m/deref (m/schema ?schema {:registry registry})))

(schema :common/single-line)
; => [:re {:bubu :lala} #"^[^\r\n]*$"]
#2021-03-0917:49bedersThanks for the help and explanation. It would be good to have some documentation around explaining instances vs. Schema. I’m still confused 🙂#2021-03-0507:41ikitommicomments welcome on how it should work.#2021-03-0508:25ordnungswidrigIs there a standard body of localized error messages for humanize?#2021-03-0508:28dharriganOnly english https://github.com/metosin/malli/blob/master/src/malli/error.cljc#2021-03-0508:28dharriganHowever, the docs do show how to add in other i18n messages#2021-03-0508:29dharriganhttps://github.com/metosin/malli#custom-error-messages#2021-03-0511:51borkdude@ikitommi What about making a plugin for malli which inspects clojure.spec specs and emits a malli schema from it? ;) Might help people migrating to malli#2021-03-0512:55ikitommi@borkdude brilliant idea. Let's do it.
#2021-03-0517:12dviramontesHi, trying to wrap my mind around how these two examples can/will differ over time when it comes to value generation
(def CDN1
  [:map
   [:images [:vector string?]]])

(def CDN2
  [:map
   [:images [:sequential string?]]])

(malli.generator/generate CDN1)
(malli.generator/generate CDN2)
-- i guess the question is - will they defer ? and if so how ? in my testing the generated values are very similar
#2021-03-0517:15borkdude@dviramontes in clojure sequential can also be a list or lazy-seq for example, not always a vector, but that should not affect equality semantics#2021-03-0517:17borkdudeit seems the generator doesn't really generate anything other than vectors at the moment (by experimentation) but it could#2021-03-0517:18dviramontesgotcha, thanks very much!#2021-03-0517:19borkdudeTIL, malli also supports keywords as predicates:
(malli.generator/generate [:sequential :int])
I prefer that syntax personally
#2021-03-0517:20borkdude
user=> (malli.core/validate :int 1)
true
#2021-03-0620:12ikitommipulling out malli schemas for defns (Vars):
(-var-schema #'+)
;[:function
; [:=> :catn :any]
; [:=> [:catn [x :any]] :any]
; [:=> [:catn [x :any] [y :any]] :any]
; [:=> [:catn [x :any] [y :any] [more [:+ :any]]] :any]]

(-var-schema #'println)
; [:=> [:catn [more [:* :any]]] :any]
#2021-03-0620:17borkdudeIs this valid syntax?
[:=> :catn :any]
#2021-03-0620:19ikitommiyes, vectors are optional if there are no childs or props, e.g. [:int] can be written as :int.#2021-03-0620:20ikitommisame for :catn. “empty sequence with named children”#2021-03-0620:20ikitommithe imp btw:
(defn -var-schema [var]
  (let [-=> (fn [as] (let [[f s [t]] (partition-by #{'&} as)
                           [fas ras rop] (cond t [f t :+], s [nil (first s) :*], :else [f])]
                       [:=> (if (or (seq fas) ras)
                              (vec (concat [:catn] (mapv (fn [a] [a :any]) fas) (when ras [[ras [rop :any]]])))
                              :catn) :any]))
        {:keys [arglists]} (meta var)
        -=>s (mapv -=> arglists)]
    (if (second arglists) (into [:function] -=>s) (first -=>s))))
#2021-03-0620:21ikitomminot sure if it’s a good idea to use :arglistsvar meta, but, works on my repl just now at least 🙂#2021-03-0620:28borkdudenot a bad idea#2021-03-0620:29borkdudesome Clojure vars use something like x* which denotes multiple xs, so the arglist is not always reliable, but in most cases it's auto-generated from the defn itself#2021-03-0712:02mike_ananev@ikitommi what is the difference between :title and :description in spec? I want to describe my config file using malli and guessing what should I use to describe every field in the config map.#2021-03-0712:09mike_ananev
(def http-server-host [:and {:title "http server host"} ne-string])
;; or
(def http-server-host [:and {:description "http server host"} ne-string])
#2021-03-0712:24ikitommi@mike1452 those are pulled from JSON Schema convention: > The `title` and `description` keywords must be strings. A “title” will preferably be short, whereas a “description” will provide a more lengthy explanation about the purpose of the data described by the schema. I would use :description there. In OpenAPI & Swagger, title is pulled as the Schema name, e.g.
[:map {:title "User", :description "Describes User of the System"} 
 [:name {:description "Name of the User"} :string] 
 [:age {:description "Age, must be >= 18} [:int {:min 18}]]]
#2021-03-0712:28mike_ananevThank you.#2021-03-0712:43ikitommipulling the return types too:
(-var-schema #'str)
;[:function
; [:=> :catn :string]
; [:=> [:catn [x :any]] :string]
; [:=> [:catn [x :any] [ys [:+ :any]]] :string]]

(-var-schema #'distinct?)
;[:function
; [:=> [:catn [x :any]] :boolean]
; [:=> [:catn [x :any] [y :any]] :boolean]
; [:=> [:catn [x :any] [y :any] [more [:+ :any]]] :boolean]]
#2021-03-0712:53borkdudeClever:
user=> (meta (second (:arglists (meta #'str))))
{:tag java.lang.String}
#2021-03-0713:01ikitommiNoticed too that the hand-written arglists seem to have weird syntax.#2021-03-0716:06joshkhloving malli so far! what's the idiomatic way of "extending" a schema? for example let's say i have a Person map with the basic attributes first-name and last-name, and then a ProfessionalPerson that is a Person with an additional required profession attribute. would i create a ProfessionalPerson definition that contains only just [:map [:profession string]] and then mu/merge it on top of the Person map [:map [:first-name string?][:last-name-string?]]?#2021-03-0716:12borkdude@joshkh Or mu/assoc#2021-03-0716:17joshkhcool, thanks borkdude. on a side note it's too bad that mu/assoc doesn't allow for & kvs like clojure.core/assoc 😄#2021-03-0716:19borkdude;)#2021-03-0716:22juhoteperiMight be possible to implement. The options parameter makes it a bit inconvenient though, we could check that if odd number of params, last param is the options, and for even number of params, no options, just kv pairs. Not sure if worth the complexity.#2021-03-0716:23juhoteperimalli.util/dissoc also takes just one key.#2021-03-0716:23borkdudeif you need more than one k/v, you can just use mu/merge :)#2021-03-0716:24borkdudeor repeat mu/assoc#2021-03-0716:24joshkhyeah, merge is fine by me. i just thought it was funny that my first instinct was to treat it like the assoc i know.#2021-03-0716:26juhoteperiYeah. Though I think it will be worth to mention the differences to clojure.core functions in the docstrings, now the util fns mention just "like clojure.core/x" but then many of them take a bit different parameters.#2021-03-0716:31joshkhperhaps. admittedly i just tried it out while completely ignoring the very clear docstring params in my editor. you can lead a horse to water but you can't make it drink 😉#2021-03-0716:33borkdudeIt's awesome that you can do this with malli btw.#2021-03-0716:37joshkhabsolutely. and the deep merging is really handy, too#2021-03-0717:34ikitommithere are at least two options to resolve the malli.util varargs issue: 1. make all utilities only work with Schema instances, e.g. no auto-coercion from schema AST => only place to pass the options would be the m/schema -> simpler, more boilerplate, 2. make a custom type/record/protocol for the options, could be the first argument in all functions, “the schema context” - easy to distinguish it from other args #2021-03-0719:47schmeedo I need to have sci for a spec like this?
[:map {:gen/fmap 'map->Point}
       [:lat [:double {:min -180.0 :max 180}]] 
       [:long [:double {:min -180.0 :max 180}]]]]
#2021-03-0719:53schmeeseems like it according to the tests: https://github.com/metosin/malli/blob/master/test/malli/generator_test.cljc#L94-L105#2021-03-0719:53schmeecan you configure malli to use the clojure compiler for eval instead of sci?#2021-03-0720:06schmeeI can’t get this to work with or without sci, can you not use namespace-qualified functions as arguments to :gen/fmap? :thinking_face:#2021-03-0720:07ikitommiyes, just don't quote and it works#2021-03-0720:07ikitommi(but, can't be deserialized if it's a fn value)#2021-03-0720:08ikitommiIf you wan't eval, PR welcome. Could be option :malli.core/evaluator a, there could be a -eval-evaualuator in malli.core#2021-03-0720:09ikitommirelevant code: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1739-L1741#2021-03-0720:12schmeeahh, I was fooled by the output of my REPL, it printed the record as a plain map (and skipped the record type)!#2021-03-0720:13schmeethen all is well, thanks for the help 🙂#2021-03-0810:03borkdude
user=> (time (m/validate [:cat [:+ [:+ [:enum 0]]] [:enum 1]] [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]) )
"Elapsed time: 0.582137 msecs"
true
user=> (time (s/valid? (s/cat :zeroes (s/+ (s/+ #{0})) :one #{1}) [0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1]))
"Elapsed time: 1265.3679 msecs"
(got the example from https://quanttype.net/posts/2021-03-06-clojure-spec-and-untrusted-input.html)
#2021-03-0810:17borkdudeAre there any examples of regexes that are slow in malli too?#2021-03-0810:24miikkaMalli's regular expression matching algorithm is fundamentally different, so as far as i know, it does not have the exponential slow down problem. Probably you could still come up with some slow examples, but I don't have insight into what is slow and what is fast.#2021-03-0810:32borkdudeBecause it doesn't have backtracking, right?#2021-03-0813:12miikkaIt does use backtracking, but see the ns docstring for discussion: https://github.com/metosin/malli/blob/master/src/malli/impl/regex.cljc#2021-03-0814:48borkdude#2021-03-0815:23arundilipanSorry for the late response @borkdude @ikitommi, I had an health emergency this week and so I wasn’t able to do anything about it#2021-03-0815:23arundilipanI did follow up with the example and it did work#2021-03-0815:23arundilipanI have a follow-up question, though#2021-03-0815:35arundilipanI’m wondering if m/decode with the above schema is supposed to produce that result, and if so, I was wondering how to solve this so that I return [1.0 "A"] from decode?#2021-03-0817:42ikitommi@arundilipan works on my machine:
(ns sample.core
  (:require [malli.core :as m]
            [malli.transform :as mt]))

(def GreaterThanZero
  [:fn {:decode/string mt/-string->double}
   '#(> % 0)])

(def TestEnum
  [:enum "A" "B"])

(m/decode
  [:catn
   [:amount GreaterThanZero]
   [:layout TestEnum]]
  ["1.0" "A"]
  mt/string-transformer)
; => [1.0 "A"]
#2021-03-0817:45ikitommiif the result doesn’t match the schema after the transformation, the top-level schema returns the original. this is a feature of the current regex impl, doesn’t support partial transformation:
(m/decode
  [:catn
   [:amount GreaterThanZero]
   [:layout TestEnum]]
  ["1.0" "C"]
  mt/string-transformer)
; => ["1.0" "C"]
#2021-03-0817:46arundilipanOh that’s it, you’re right#2021-03-0817:47arundilipanIn that case I guess the thing to do would be do use something like edn/read-string instead of decode? I’d like to make sure that the number is parsed regardless of the result of the rest of the :catn #2021-03-0817:48ikitommior, you could use non-regex schema like :tuple:
(m/decode
  [:tuple GreaterThanZero TestEnum]
  ["1.0" "C"]
  mt/string-transformer)
; => [1.0 "C"]
#2021-03-0817:49ikitommino :tuplen atm 😉#2021-03-0817:50ikitommiyou could write issue about the partial decoding with regex?#2021-03-0817:50ikitommi(might be easy to fix)#2021-03-0817:51arundilipanYea i’ll make an issue on the repo#2021-03-0817:51arundilipanRight now I’m trying to use the :catn variants because of the coercion into a map#2021-03-0817:55arundilipanDone!#2021-03-0918:06raymcdermottI'm trying to take a simple map like this an make it consumable by reitit-swagger#2021-03-0918:07raymcdermott
(def Org
  [:map
   [:id Id]
   [:ref string?]])
#2021-03-0918:08raymcdermottafter swagger/transform I get#2021-03-0918:08raymcdermott
{:type "object",
 :properties {:id {:type "string",
                   :pattern #"^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$"},
              :ref {:type "string"}},
 :required [:id :ref]}
#2021-03-0918:11raymcdermottnot sure how / where to plug this in to the reitit.swagger structure#2021-03-0918:12raymcdermottlooking at the example in reitit I see this#2021-03-0918:12raymcdermott
["/plus"
         {:get {:summary "plus with malli query parameters"
                :parameters {:query [:map [:x int?] [:y int?]]}
                :responses {200 {:body [:map [:total int?]]}}
                :handler (fn [{{{:keys [x y]} :query} :parameters}]
                           {:status 200
                            :body {:total (+ x y)}})}
          :post {:summary "plus with malli body parameters"
                 :parameters {:body [:map [:x int?] [:y int?]]}
                 :responses {200 {:body [:map [:total int?]]}}
                 :handler (fn [{{{:keys [x y]} :body} :parameters}]
                            {:status 200
                             :body {:total (+ x y)}})}}]
#2021-03-0918:14raymcdermottI can see that I should decorate the Org with some swagger but still, I'm being dumb cos I can't see how to smash everything together#2021-03-0918:15raymcdermottany clues if you have done this already would be appreciated#2021-03-0918:48ikitommi@raymcdermott if the swgger-transform output is ok, just use it like {:parameters {:body Org}} and it works. swagger-transformation is done in the reitit.swagger/create-swagger-handler code for all parameters & response schemas automatically, it looks the effective :coercion for a route and asks it to transform the stuff.#2021-03-0918:49ikitommihere’s the code: https://github.com/metosin/reitit/blob/master/modules/reitit-swagger/src/reitit/swagger.cljc#L69-L109#2021-03-0918:49ikitommiyou can add stuff with :swagger namespace or key:
(def Org
  [:map {:title "Org"}
   [:id {:swagger/description "id"} Id]
   [:ref {:swagger {:default "kikka"}} string?]])
#2021-03-0919:31raymcdermottok - great, I'll give a go. Thanks @ikitommi#2021-03-0919:51raymcdermottI can report success 🙏:skin-tone-3:#2021-03-1011:11Adam HelinsI have written a WebAssembly decompiler/compiler and I would like to "spec" the WASM intermediary representation I am using. After a long while of avoiding clojure.spec, I've tried using it for that purpose but it's just too hard, too limited. So I am trying out Malli. Given how flexible it is, I have high hopes. But I am a Malli noob so here I am. My problem is actually a common one: a data schema were parts of the schema are references ("pointers") to other parts of the schema. Validation is easy but generation is hard. Spec makes that extra hard by forcing a global registry. Let's stick to this WASM example to have something concrete to work with. There exists a type section (map of type index to type) and a function section (map of function index to function where a function holds an existing type index). Generating a valid representation in pseudo-code would look like: 1. Generate a type section where type index is a nat? 2. Mutate registry so that type index becomes an enum of generated type index 3. Generate a function section , now assured that generated type index exist Solution A. I guess one way would be to do exactly that. Generating things in the right order, step by step, and keep on doing that while successively building a local registry based on what has been previously generated. I am unsure how practical that would be for a more complex example. Solution B. Another way would be to have one schema with a "local mutable registry" and declaring custom generators along the way which would mutate this "local mutable registry" based on what they generate. However I understand that local registries must be concrete maps and do not allow this. On the longer term, I am not sure this would be more pratical than Solution A and it certainly is not very functional. Any better, idiomatic way?#2021-03-1011:27Adam HelinsNote regarding Solution B: Unless I am doing it wrong, I cannot manage to use a mutable registry in a local way (ie. using it as an explicit :registry argument throws :malli.core/invalid-schema {:schema :map})#2021-03-1013:53ikitommi@adam678 Sounds really interesting! I can’t recall why the local registries only support maps. Would be most likely a small change it to support the Registry#2021-03-1013:55ikitommione option woud be to just collect stuff to a mutable registry or an custom atom, use it via (m/schema x {:registry registry}). when everything is collected and if the schemas are serializable, you could writen them into one local (immutable) registry.#2021-03-1014:14nilernThere was a similar PR recently https://github.com/metosin/malli/pull/337#2021-03-1014:29jjttjjI'm wondering what the "limits" of the malli value transformations should be philosophically. If I'm working with an external api that represents dates as strings, but I want to represent them as Instants in my app, is that a valid use for transformers? What about if the external api provides a nested "address" map that I want to eventually turn into an Address record type? It wouldn't necessarily be used as a two way transformation very much, I wouldn't necessarily be sending my addresses back to their api and need to encode them again as strings. Is this just a separate problem, and I should just use a function to get things into my domain records/types?#2021-03-1014:30juhoteperiCoercing stings to proper types at least is fine use. Similar to coercing json or path or query string parameters to booleans, dates etc.#2021-03-1014:31nilernWe do those sorts of things all the time, also with Schema and Spec#2021-03-1014:34jjttjjSo it's good to use malli to get from string all the way to our full on domain objects?#2021-03-1014:38ikitommisure. If it like looks complex, then move the transformation out. I haven't seen a always valid limit. string->map map->record both perfect cases.#2021-03-1014:39jjttjjby "move the transformation out" you mean separate it into a different transformer?#2021-03-1014:40ikitommiwrote a while back this generic nonsense (on spec): > Domain-specific data-macros are cool, but add some complexity due to the inversion of control. For just this reason, we have pulled out Schema-based domain coercions from some of our client projects. Use them wisely.#2021-03-1014:40ikitommihttps://www.metosin.fi/blog/clojure-spec-as-a-runtime-transformation-engine/#data-macros#2021-03-1014:40jjttjjThanks!#2021-03-1014:42ikitommiby moving out I meant into a separate function outside of schemas, e.g. (external->internal data) thing.#2021-03-1014:45jjttjjAnd then do you call that from a transformer still? or do you mean the data goes from
json-string-> clojure data -> malli transformers -> external->internal
#2021-03-1015:42ikitommi• optimal: json-string + malli -> internal • current good practise: json-string -> EDN -> malli transformers -> internal • if the internal->external is complex. uses external data etc: json-string -> EDN -> malli transformers -> external->internal#2021-03-1015:44ikitommi(did a spike on deriving jackson-decoder from malli schema, but nothing production grade atm, in theory, shoud be much faster)#2021-03-1015:51jjttjjthat makes sense, thanks again!#2021-03-1015:09Adam Helins@ikitommi All right, thanks, I got a sense of where to start. As a beginner I was mostly troubled by the fact that local registries can be only map-based. Since you are not sure this is intended, I took the liberty of opening an issue: https://github.com/metosin/malli/issues/389#2021-03-1018:36EdHi. If I have a :dispatch multi-schema, is there an easy way to put a catch-all else clause in the matches?#2021-03-1019:00ikitommi@l0st3d not atm, but could, does [:or [:multi …] :default]work for you?#2021-03-1019:00ikitommimaybe there could be a :malli/default branch?#2021-03-1019:27EdI think I've worked out how to write what I was trying to write in a different way (using :or, but I needed an exclusionary condition instead of :default), but it might be a useful feature to add ... I quite like the idea of :malli/default. Partly because of the parallel to multimethods. Maybe there's a clean way of overriding the keyword?#2021-03-1019:32Edmy exclusion clause in the second branch of the or is a bit big, but cos it's all data it's ok to write a function to produce the branches 😉#2021-03-1105:14ikitommi@l0st3d & everyone else, opinions on :m/default or :malli/default?
[:multi {:dispatch :type}
 ["object" [:map-of :keyword :string]]
 [:m/default :string]]
vs:
[:multi {:dispatch :type}
 ["object" [:map-of :keyword :string]]
 [:malli/default :string]]
… same could be used for extra-keys in maps, e.g. :m/extra vs :malli/extra. Personally fond of the :m.
#2021-03-1105:14ikitommihttps://github.com/metosin/malli/pull/391#2021-03-1105:15ikitommi
(def valid?
  (m/validator
    [:multi {:dispatch :type}
     ["object" [:map-of :keyword :string]]
     [:m/default :string]]))

(valid? {:type "object", :key "1", :value "100"})
; => true

(valid? "SUCCESS!")
; => true

(valid? :failure)
; => false
#2021-03-1108:37EdThat would definitely solve my problem and make my code easier to read ... it get's my vote ... many thanks for the impressively fast feedback.#2021-03-1108:41EdAlthough maybe ::m/default might be better#2021-03-1108:41EdAh ... I see someone has already commented to that effect on the pr 😉#2021-03-1110:40Adam HelinsHehe, sometimes I dream about creating a sub-community of "Clojurists against too much abbreviation"#2021-03-1110:48Ed😉 ... I think it's not about abbreviation, but scope ... :m/ is a public namespace that may be being used by an application and may well be a target in that multi dispatch, whereas ::m/ is scoped to malli.core and therefore shouldn't have meaning in your application - so it's fine to use as a default ... I think where clojure pushes back against brevity it tends to be for reasons of applicability in different contexts - which makes it a more generally useful language#2021-03-1114:33armedHello, how to attach custom error message to seqexp? For example:
(malli.error/humanize
  (malli/explain
   [:map
    [:items
     [:+
      {:error/message "Items must not be empty"}
      [:map [:foo string?
             :bar int?]]]]]
   {:items []}))
#2021-03-1117:46shanlooks like the :error/message is in the wrong place:
(me/humanize
 (m/explain
  [:map
   [:items
    [:+
     [:map {:error/message "Items must not be empty"}
      [:foo string?
       :bar int?]]]]]
  {:items []}));; => {:items [["Items must not be empty"]]}
#2021-03-1118:55armedThanks, thats interesting. My code works if I replace :+ with :vector#2021-03-1118:57armedAnd if I put error inside :map, then all other map validation errors replaced with that one, even ‘missing key’ is becomes ‘items must not be empty’. #2021-03-1114:34armedI get {:items [["unknown error"]]} error#2021-03-1117:59emccueHow would I represent values like this#2021-03-1117:59emccue
:keyword-a
:keyword-b
{:map "with" :some 'shape}
#2021-03-1118:00emccue(in the same schema)#2021-03-1118:02emccueat least with the online tool, alt doesn't seem to do what I would expect#2021-03-1118:03emccue#2021-03-1118:33ikitommi@emccue try https://malli.io/?value=%5B%3Aa%20%3Ab%20%7B%3Aname%20%22kikka%22%7D%5D&amp;schema=%5B%3Avector%20%0A%20%5B%3Aor%20%0A%20%20%5B%3Aenum%20%3Aa%20%3Ab%5D%0A%20%20%5B%3Amap%20%5B%3Aname%20%3Astring%5D%5D%5D%5D#2021-03-1118:34ikitommithe http://malli.io is not updated for 0.3.0, so no :alt there yet. PR welcome to update deps#2021-03-1118:38emccuewould alt work in that case?#2021-03-1118:43nilern:alt is for sequences only#2021-03-1118:47nilern[:alt [:= :a] [:= :b]] matches [:a] and (:b) but not :a#2021-03-1119:08emccueinteresting that [:enum :a :b] is different than [:or [:= :a] [:= :b]]#2021-03-1119:23ikitommidifferent, how?#2021-03-1120:06emccuelike, they are inferred differently#2021-03-1200:42emccue(looped back to this, i'm probably wrong and got confused by the "inferred schema" part and also my bad memory)#2021-03-1119:19ikitommiupdated http://malli.io, with few sequence samples (args & hiccup)#2021-03-1119:20ikitommihttps://malli.io/?value=%5B%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D%5D&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D#2021-03-1119:20ikitommihttps://malli.io/?value=%5B%22-server%22%20%22foo%22%20%22-verbose%22%20true%20%22-user%22%20%22joe%22%5D&amp;schema=%5B%3A*%20%5B%3Acatn%20%5B%3Aprop%20string%3F%5D%20%5B%3Aval%20%5B%3Aaltn%20%5B%3As%20string%3F%5D%20%5B%3Ab%20boolean%3F%5D%5D%5D%5D%5D#2021-03-1119:21ikitommithe Hiccup in JSON Schema looks fishy:
{:$ref "#/definitions/hiccup", :definitions {"hiccup" {}}}
#2021-03-1119:22ikitommibut, there are no sequence schemas there, so i guess it’s the best we can do.#2021-03-1212:59raymcdermottI want to protect the swagger doc built from the malli data model using buddy. I have seen the reitit examples which protect routes but it's not clear how this integrates into the Swagger model ... I will also x post to reitit#2021-03-1215:52raymcdermott[ thanks for the answer by @ikitommi in #reitit ]#2021-03-1218:39ikitommimany ways of stripping extra keys from maps: https://www.reddit.com/r/Clojure/comments/m3hx1e/how_do_i_walk_a_complex_map_and_remove_keys_based/#2021-03-1221:36borkdudeJust had a thought. Can malli be used in places where currently meander or matchete (https://github.com/xapix-io/matchete) are used? Although this works, it is a bit cumbersome:
user=> (m/validate [:cat [:enum 1] :any [:enum 3]] '[1 2 3])
true
#2021-03-1221:39borkdudeAnd how could I get the destructured value, after validation?
user=> (m/explain [:catn [:a [:enum 1]] [:b :any] [:c [:enum 3]]] '[1 2 3])
nil
#2021-03-1222:29borkdudeoh of course, parse facepalm
user=> (m/parse [:catn [:a [:enum 1]] [:b :any] [:c [:enum 3]]] '[1 2 3])
{:a 1, :b 2, :c 3}
#2021-03-1222:49borkdudeIs there a way to indicate that you want to ignore a certain binding in the parsed output?#2021-03-1307:44ikitomminot at the moment. issue welcome.#2021-03-1222:57borkdudeI hacked it like this: https://gist.github.com/borkdude/26906ee15585ed5e1b7a8eda4cc1ee18#2021-03-1307:45ikitommi::m/default for :multi merged in master:
(def valid?
  (m/validator
    [:multi {:dispatch :type}
     ["object" [:map-of :keyword :string]]
     [::m/default :string]]))

(valid? {:type "object", :key "1", :value "100"})
; => true

(valid? "SUCCESS!")
; => true

(valid? :failure)
; => false
#2021-03-1307:50ikitommiMaybe the same could be used for extra keys in maps? (https://github.com/metosin/malli/issues/43)#2021-03-1309:53borkdude@ikitommi I was trying (for fun) to write a core.match like thing with malli. So {:a ?x :b 1} would match on {:a 2 :b 1} and parsing would give you back {?x 1}. This is currently a bit difficult since you cannot change how keys are named in the parsed output from map schemas, can you?#2021-03-1309:54borkdudeSo I would generate a schema like {:b [:= 1] :a :any} but then the parsed output loses the name ?x#2021-03-1310:07borkdudeCan I influence how things get parsed using an extra predicate similar to s/and in spec?
(m/parse [:map [:a [:and :any
                         [:fn (fn [x]
                                ['x? x])]]]]
              {:a 1})
#2021-03-1310:13ikitomminot atm, parsing could have it's own property key for this, e.g. [:map {:parse ...} ...]. Internally, could be interceptors, so one can do easily pre, post & schema-based parsing with that.#2021-03-1310:13ikitommiWe discussed with @nilern about combining internally parsing, explain and transform. They are now mostly (optimized) duplicates of each other.#2021-03-1310:15ikitommimap keys -> could done with same mechanism. Add custom parse tags to keys as properties and hook a post-parse fn to rename the keys. Or do, whatever.#2021-03-1311:04ikitommiported the plumatic-style inline schemas for 0.3.0, it kinda works, but no tests and not happy with the original (string-based) error reporting. renamed to ns to malli.experimental.schema as it might not be part of the malli core library. any thoughts on this? add tests, cleanup and ship as experimental add-on?#2021-03-1311:04ikitommihttps://github.com/metosin/malli/pull/305#2021-03-1311:06ikitommithe defn now emits a function schema into malli function registry and it can be configured to validate always, never or based on a dynamic var - at runtime.#2021-03-1414:18vemvReplied in https://github.com/metosin/malli/issues/125 . Btw I hope I'm not pestering too much - I simply try to make an educated attempt at improving perceived problems (which can last long - for example I use Schema at work and it kinda hurts to use an outdated tech that turned out to not be the best bet)#2021-03-1313:03lmergenwhat is the validator i can use for 'any collection, no matter the type' ? i don't care whether it's a sequence or vector or set, just that it's a collection. what can i use for this? i wasn't able to find it#2021-03-1314:26ikitommi@lmergen maybe coll?#2021-03-1314:27lmergeni can use that like [:coll pos-int?] ?#2021-03-1314:30lmergentrying to figure out the best way to do this 🙂#2021-03-1314:31ikitommisadly, it's just the core predicate, so no child type checking. There is no :coll atm. Would be a oneliner to add#2021-03-1314:31ikitommiSee https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1858#2021-03-1314:32lmergenis there a specific reason why this is not in malli itself? as in, if i would send a PR to add this, would that be useful?#2021-03-1314:32ikitommino-one has asked, PR welcome (tests included)!#2021-03-1314:33lmergenof course 👍#2021-03-1314:33ikitommiit spans to generator, json schema, human errors etc. But, good examples :)#2021-03-1314:33lmergenthanks, i'll see what i can do#2021-03-1314:33lmergenyes it touches a lot of surface#2021-03-1314:44juhotepericoll? also matches maps, that has some effects on JSON-Schema implementation at least#2021-03-1315:19ikitommigood point. I guess there is not a predicate for non-map collection.#2021-03-1315:24nilernI imagine maps also having keys can lead to all sorts of confusion when implementing [:coll pos-int?]#2021-03-1315:24nilernWhat about :sequential?#2021-03-1315:25nilernOh that does not take sets#2021-03-1315:31nilernCan always do [:or [:sequential pos-int?] [:set pos-int?]]#2021-03-1315:33nilernThat does not support seqs though... and now I see nothing else does either.#2021-03-1315:37nilernMade some issues about that https://github.com/metosin/malli/issues/393 https://github.com/metosin/malli/issues/394#2021-03-1506:28fmnCurrently, is it possible to do this with parse ?
(s/def ::name-or-id (s/or :name string?
                          :id   int?))
(s/conform ::name-or-id "abc")
;;=> [:name "abc"]
(s/conform ::name-or-id 100)
;;=> [:id 100]
#2021-03-1508:09ikitommi@funyako.funyao156 try [:orn [:name string?] [:id int?]]. There are named-branch variants for :or, :alt and :cat with +n in the name.#2021-03-1511:06fmn@ikitommi Thanks!#2021-03-1618:01raymcdermott@ikitommi trying to get the graphviz example working with dorothy but it's all a string. What do you use to render it?#2021-03-1618:12ikitommi@raymcdermott try println#2021-03-1618:54raymcdermottI meant to a PNG or whatever ...#2021-03-1619:39ikitommioh, that… 🙂 just dot from command line or the online version, e.g. https://sketchviz.com/#2021-03-1619:40ikitommilove the sketches#2021-03-1619:46ikitommibut, with dorothy, this seems to work:
(require '[dorothy.core :as d])

(->> (md/transform Order) (d/show!))
#2021-03-1620:26raymcdermottok ... I was holding it wrong#2021-03-1621:23raymcdermottalso m/walk ... thank you!!#2021-03-1622:11raymcdermotthaving said that#2021-03-1622:11raymcdermotthow do I use it to drop or add in Swagger attributes?#2021-03-1622:12raymcdermott
(def Org-Ref
  [:map {:title "Organisation name"}
   [:ref {:swagger/description "Reference to the organisation"
          :swagger/example     "Acme floor polish, Houston TX"} Name]])
#2021-03-1622:12raymcdermottI tried mu/update-properties but that only seems to operate at the top level#2021-03-1622:13raymcdermott[ cos they clutter up the graphics 🙂 ]#2021-03-1706:26ikitommi@raymcdermott you can walk + update-properties, but if you want to walk also the map-entries (not just map values), you should paramertise the walk to walk those too. same for :refs.#2021-03-1706:26ikitommisome tests on walking here: https://github.com/metosin/malli/blob/master/test/malli/util_test.cljc#L672-L795#2021-03-1706:27ikitommie.g. ::m/walk-refs & walk-schema-refs & ::m/walk-entry-vals.#2021-03-1706:28ikitommihopefully not too complex, balancing between “you can do anything with this” and “should be easy to do the simple stuff”#2021-03-1706:29ikitommialso, the dot-printer could have a option with schema->schema function to manipulate the schemas before printing?#2021-03-1709:04raymcdermottI will have to consider that once I get it all to work 🙂#2021-03-1717:17raymcdermottI have read the tests and am struggling to see how they help me to drop the swagger entries. I'm not even sure how to address them .... I know they are the the map in slot 1 of the key :ref but can see how to access that. None of them seem to show how to manipulate entries. And, I know it's my limitation but going to the schema processing itself is quite complex code to read out in my head.#2021-03-1717:20raymcdermottI was thinking that - since they are only things defined with maps, I could find a way to drop maps like I would do for some type with walk#2021-03-1717:39ikitommiI'll try it out#2021-03-1718:09ikitommioh, no easy way for that. will add something.#2021-03-1718:35raymcdermottthanks @U055NJ5CC#2021-03-1802:57yuhanHi, I'm just working through the Readme tutorial on the latest 0.3.0 and found that this example failed:
(m/validate
  [:map
   ["status" [:enum "ok"]]
   [1 any?]
   [nil any?]
   [::a string?]]
  {"status" "ok"
   1 'number
   nil :yay
   ::a "properly awesome"})
; => true
Instead I get an exception:
1. Unhandled clojure.lang.ExceptionInfo
   :malli.core/naked-keys-not-supported nil
   {:type :malli.core/naked-keys-not-supported, :data nil}
#2021-03-1803:00yuhanIs this a regression or API change? I couldn't find references to "naked keys" apart from internal impl code#2021-03-1805:29ikitommi@qythium it was a regression, fixed in master#2021-03-1808:19HankstenbergIs there a way to filter data by a schema? So if I have data of the form {:a 1 :b 2} and a schema of the form [:map [:a int?]] can I "apply" the schema to the data in order to get {:a 1}? All I can think of right now is to use m/explain then parse the errors.#2021-03-1809:01ikitommi@raymcdermott would this be ok:
(def Org-Ref
  [:map {:title "Organisation name"}
   [:ref {:swagger/description "Reference to the organisation"
          :swagger/example "Acme floor polish, Houston TX"} :string]
   [:kikka [:string {:swagger {:title "kukka"}}]]])

(defn remove-swagger-keys [p]
  (not-empty (apply dissoc p (into #{:swagger} (->> p (keys) (filter (comp #{:swagger} keyword namespace)))))))

(defn walk-properties [schema f]
  (m/walk
    schema
    (fn [s _ c _]
      (m/into-schema
        (m/-parent s)
        (f (m/-properties s))
        (cond->> c (m/entries s) (map (fn [[k p s]] [k (f p) (first (m/children s))])))
        (m/options s)))
    {::m/walk-entry-vals true}))

(walk-properties Org-Ref remove-swagger-keys)
;[:map {:title "Organisation name"} 
; [:ref :string] 
; [:kikka :string]]
#2021-03-1809:02ikitommie.g. walk the entrys, un-walk on the way back. apply f on all properties (entrys & schemas)#2021-03-1809:03ikitommi@roseneck you can transform the value using strip-extra-keys-transformer:
(m/decode [:map [:a int?]] {:a 1, :b 2} (mt/strip-extra-keys-transformer))
; => {:a 1}
#2021-03-1809:09Hankstenberg@ikitommi perfect, thank you very much!#2021-03-1809:50raymcdermottyes @ikitommi that's very elegant#2021-03-1810:05robert-stuttafordis it possible to introspect the malli schema from inside an :error/fn fn ? i.e. i want to (first (m/children schema)) so i can print out the enum values in the error message#2021-03-1810:34ikitommi@robert-stuttaford sure, the fn takes the explain error map as argument, it has the :schema key.#2021-03-1810:35ikitommithere are lot of examples in malli.error#2021-03-1810:38ikitommihttps://github.com/metosin/malli/blob/master/src/malli/error.cljc#L75-L80#2021-03-1811:02robert-stuttafordwhen i try this, i get m/children isn't a thing, because it's SCI that's running this code#2021-03-1811:02robert-stuttaford(thank you for your quick response)#2021-03-1811:05robert-stuttaford
Execution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:51).
Could not resolve symbol: m/children
#2021-03-1811:05robert-stuttafordis there a trick to get it to see malli?#2021-03-1811:08borkdude@robert-stuttaford just out of curiosity: what is your use case for malli + SCI?#2021-03-1811:08borkdudemalli could make this namespace available inside of sci#2021-03-1811:08ikitommithere is ::m/sci-options to override the bindings. The default bindings are:
(defn -default-sci-options []
  {:preset :termination-safe
   :bindings {'m/properties properties
              'm/type type
              'm/children children
              'm/entries entries}})
have the sci-options changed? don’t seem to work anymre
#2021-03-1811:09ikitommi:termination-safe is removed at least#2021-03-1811:09borkdudethe options have not been changed, but :bindings are only valid within the user namespace, always have been. it's better to use explicit :namespaces#2021-03-1811:09robert-stuttafordhonestly i'm using sci because malli is#2021-03-1811:09borkdudeyes :termination-safe has been removed for a while already, also documented in release notes#2021-03-1811:10robert-stuttafordall i'm doing is writing malli specs at the repl with :error/fn and i ran into an error 'sci not available', so i put it on the CP and onward i went#2021-03-1811:10ikitommi@robert-stuttaford if you don’t need the seriaization thing, just pass a real function.#2021-03-1811:10robert-stuttafordoh man. the fn is quoted. shit. sorry for the noise, fellas#2021-03-1811:10ikitommi🙂#2021-03-1811:10borkdudethat's what I thought :) maybe the error message should be : use sci for serialized schemas#2021-03-1811:10borkdudeor something#2021-03-1811:11ikitommiall properties which have functions as values use the malli.eval, which uses sci as default for quoted code.#2021-03-1811:12ikitommie.g.`:gen/fmap '(partial str "kikka_")`#2021-03-1811:12ikitommibut, what is the right way to bind those m/children into sci via options?#2021-03-1811:13borkdude{:namespaces {'malli.core {'children m/children}}}#2021-03-1811:15ikitommidoesn’t work either.#2021-03-1811:15ikitommi
(defn -default-sci-options []
  {:namespaces {'malli.core {'properties properties
                             'type type
                             'children children
                             'entries entries}}})
#2021-03-1811:15borkdudedoesn't work = which error?#2021-03-1811:15ikitommi
Execution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:50).
Could not resolve symbol: malli.core/chidren [at line 1, column 10]
#2021-03-1811:16ikitommi
(defn evaluator [options fail!]
  (let [eval-string* (dynaload/dynaload 'sci.core/eval-string* {:default nil})
        init (dynaload/dynaload 'sci.core/init {:default nil})
        fork (dynaload/dynaload 'sci.core/fork {:default nil})]
    (fn [] (if (and @eval-string* @init @fork)
             (let [ctx (init options)]
               (fn eval [s] (eval-string* (fork ctx) (str s))))
             fail!))))
#2021-03-1811:16borkdude"malli.core/chidren" <- typo?#2021-03-1811:16ikitommi:face_palm:#2021-03-1811:16ikitommithanks, works like a charm#2021-03-1811:17ikitommiwhat about binding m -> malli.core for not breaking things?#2021-03-1811:29borkdudeyou can manually insert a (require '[malli.core :as m]) to ensure this works#2021-03-1811:30borkdudeor (alias 'm 'malli.core)#2021-03-1811:33ikitommilike:
(defn evaluator [options fail!]
  (let [eval-string* (dynaload/dynaload 'sci.core/eval-string* {:default nil})
        init (dynaload/dynaload 'sci.core/init {:default nil})
        fork (dynaload/dynaload 'sci.core/fork {:default nil})]
    (fn [] (if (and @eval-string* @init @fork)
             (let [ctx (init options)]
               (eval-string* ctx "(alias 'm 'malli.core)")
               (fn eval [s] (eval-string* (fork ctx) (str s))))
             fail!))))
#2021-03-1811:33ikitommiseems to work#2021-03-1811:34ikitommi
((m/eval 'm/type) :int)
; => :int
#2021-03-1811:34ikitommi🙇#2021-03-1810:52euccastrohow do I transform the keys of a schema? e.g., I have [:map [:a-k string?] [:b-k string?]] and I want to derive a schema that is the same but with snake-case keys: [:map [:a_k string?] [:b_k string?]]#2021-03-1811:21ikitommi@euccastro try m/walk with m/schema-walker check that the schema is a :map and recreate the children.#2021-03-1811:25euccastrothank you!#2021-03-1811:47robert-stuttafordis the best way to specify a literal value to use a single-item enum?#2021-03-1811:49delaguardo[:= value]#2021-03-1811:50robert-stuttafordnoice!#2021-03-1812:15borkdudeTried to use this to build a core.match like this ;) https://gist.github.com/borkdude/26906ee15585ed5e1b7a8eda4cc1ee18#2021-03-2207:03robert-stuttafordnice man#2021-03-1816:35danielnealdumb question, how do you get a schema from the registry by its key#2021-03-1816:35danielnealI was thinking (malli/-schema malli/default-registry ::sq/sqid)#2021-03-1816:35danielnealbut -schema is private#2021-03-1816:37emccue
[:map
    [:field-a string?]
    [:field-b [:one-of 
               [:map [:status [:= :not-asked]]]
               [:map [:status [:= :loading]]]
               [:map [:status [:= :failed]]]
               [:map [:status [:= :success]
                      :value  [:vector [:map [:id int?]]]]]]]
    [:field-c [:one-of
                [:map [:status [:= :not-asked]]]
                [:map [:status [:= :loading]]]
                [:map [:status [:= :failed]]]
                [:map [:status [:= :success]
                       :value  [:vector [:map [:id int?]]]]]]]
    
    [:field-d [:set [:map [:id int?]]]]]
#2021-03-1816:37emccuewhat would the idiomatic way to represent this be?#2021-03-1816:39emccuetrying on the playground it doesn't say its wrong#2021-03-1816:39emccue#2021-03-1816:39emccuebut it also doesn't produce any sample values#2021-03-1816:40ikitommi@emccue there is no one-of, you can use :or#2021-03-1816:41emccueoh.#2021-03-1816:41emccueyep that did it#2021-03-1816:42ikitommicould also be a :multi dispatching on :type .#2021-03-1816:44ikitommi@danieleneal try (m/deref (m/schema ::sq/said))#2021-03-1816:45emccueWhat is the benefit of multi schemas over explicit listing like that?#2021-03-1816:45ikitommi(m/deref ::sq/said) might work too.#2021-03-1816:47ikitommimight not be big difference, but performance. dispatch does one lookup to find the correct schema, :or does linear scan over all.#2021-03-1816:49emccueI think last question for now - what would be the best way to reuse a structure like this#2021-03-1816:49emccuejust a function that takes in the success value schema and returns the whole thing?#2021-03-1816:56danielneal@ikitommi, (m/deref ::sq/sqid) works thanks :thumbsup:#2021-03-1817:27danielnealis there a way of getting the schema walker to deref while walking? I've got a schema which is like [:map [:some-key :some-schema] [:another-key :another-schema]] where :some-schema and :another-schema are in the registry. I want to transform all the keys to snake case, but the schema walker doesn't descend into schema references.#2021-03-1817:35ikitommi@danieleneal see: > e.g. ::m/walk-refs & ::m/walk-schema-refs & ::m/walk-entry-vals.#2021-03-1817:36ikitommiwalking respects those options, can't recall what does what. Please try, documentation PR welcome#2021-03-1817:36ikitommiI recall those are recursion safe#2021-03-1817:37ikitommie.g. stop of first deref of already walked reference#2021-03-1817:38danielnealoh cool#2021-03-1817:38danielnealthanks again!!!#2021-03-1821:21ikitommi@emccue one way to reuse is to use local registry:
[:map {:registry {:user/success [:map
                                 [:status [:= :success]]
                                 [:value [:vector [:map [:id int?]]]]]
                  :user/default [:map 
                                 [:status [:enum :not-asked :loading :failed]]]
                  :user/field [:multi {:dispatch :status}
                               [:success :user/success]
                               [:malli.core/default :user/default]]}}
 [:field-a string?]
 [:field-b :user/field]
 [:field-c :user/field]
 [:field-d [:set [:map [:id int?]]]]]
#2021-03-1821:21ikitommihttps://malli.io/?value=%7B%3Afield-a%20%22kikka%22%2C%0A%20%3Afield-b%20%7B%3Astatus%20%3Aloading%7D%2C%0A%20%3Afield-c%20%7B%3Astatus%20%3Afailed%7D%2C%0A%20%3Afield-d%20%23%7B%7B%3Aid%209807%7D%7D%7D&amp;schema=%5B%3Amap%20%7B%3Aregistry%20%7B%3Auser%2Fsuccess%20%5B%3Amap%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Astatus%20%5B%3A%3D%20%3Asuccess%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Avalue%20%5B%3Avector%20%5B%3Amap%20%5B%3Aid%20int%3F%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Fdefault%20%5B%3Amap%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Astatus%20%5B%3Aenum%20%3Anot-asked%20%3Aloading%20%3Afailed%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%3Auser%2Ffield%20%5B%3Amulti%20%7B%3Adispatch%20%3Astatus%7D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Asuccess%20%3Auser%2Fsuccess%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Amalli.core%2Fdefault%20%3Auser%2Fdefault%5D%5D%7D%7D%0A%20%5B%3Afield-a%20string%3F%5D%0A%20%5B%3Afield-b%20%3Auser%2Ffield%5D%0A%20%5B%3Afield-c%20%3Auser%2Ffield%5D%0A%20%5B%3Afield-d%20%5B%3Aset%20%5B%3Amap%20%5B%3Aid%20int%3F%5D%5D%5D%5D%5D#2021-03-1821:22ikitommicould be of course global registry too.#2021-03-1821:23emccueI mean like, this is analagous to a typed enum from other langs#2021-03-1821:23emccueso i would in my brain go#2021-03-1821:23emccue(remote-data [:map [:id int?]])#2021-03-1821:23emccueand reuse the pattern#2021-03-1821:24ikitommioh, sure. it’s just data, so a function liike that is the way to do it.#2021-03-1821:29emccuebut by the same token, if i have a spec like this in a namespace#2021-03-1821:29emccue
(def user [:map [:name string?]])
#2021-03-1821:29emccueI shouldn't use it like this#2021-03-1821:30emccue
(def school [:vector other.ns/user])
#2021-03-1821:30emccueright?#2021-03-1821:30emccuebecause then it will get "flattened" and the errors won't be as good#2021-03-1821:32nilernThat's how we used to do it with Plumatic Schema#2021-03-1821:35nilernWe wanted to enable that style too, you may give up some serialization benefits but get to use normal defs etc.#2021-03-1821:40nilernThe registry names don't play any role in e.g. explainer actually#2021-03-1821:49emccueso then a local registry is basically equivalent to a let?#2021-03-1821:49emccue(but "runs" in the schema?)#2021-03-1903:12willierspec->malli was mentioned here a while ago. is anyone aware if such a thing exists?#2021-03-1906:03ikitommi@emccue currently it’s a let. Could be something else too. What kind of better errors would you expect? The reference information is available over there, just not used I guess in explain:
(m/explain
  [:map {:registry {"ID" :int}}
   [:id "ID"]]
  {:id "123"})
;{:schema [:map {:registry {"ID" :int}} [:id "ID"]],
; :value {:id "123"},
; :errors (#Error{:path [:id], :in [:id], :schema :int, :value "123"})}
#2021-03-1906:03ikitommi@willier not yet. interested in doing? 😉#2021-03-1908:43willierhmm, probably need a black belt in macro-fu to do this job#2021-03-1908:44nilernOr a spec-tools belt?#2021-03-1908:56borkdude@willier you can inspect specs at runtime. spec-tools has something called a walker and coax is a library from exoscale to do coercion based on specs, I think they are doing something similar#2021-03-1908:58ikitommispec-tools has both walker and a visitor 🙂 visitor is the right tool for the job i believe: https://cljdoc.org/d/metosin/spec-tools/0.10.5/doc/spec-visitor#2021-03-1908:58ikitommijson schma with it: https://cljdoc.org/d/metosin/spec-tools/0.10.5/doc/json-schemas#2021-03-1909:00willieroh interesting... i will have a look into it - thanks!#2021-03-1909:01ikitommialso, there are progression tests in spec-tools for broken forms: if the core would be fixed: https://github.com/metosin/spec-tools/blob/master/test/cljc/spec_tools/visitor_all_test.cljc#2021-03-1909:03ikitommioh, no more, tests disabled and the last form is fixed. great!#2021-03-1909:43danielnealIf you have a map that references other schemas, like this [:map ::some/ref ::some-other/ref] how do you make ::some/ref and ::some-other/ref optional?#2021-03-1909:47danielnealI tried [:and {:optional true} ::some/ref] but that was invalid#2021-03-1909:49nilernYou just have to use the sugar-free [:map [::some/ref {:optional true} ::some/ref]] You could get clever and [:map (#(vector % {:optional true} %) ::some/ref)] or name and reuse that little fn#2021-03-1909:52nilern[:map ::some/ref] is just some sugar in :map, we don't have "first-class properties" beyond that. And :optional is specific to map entries while :maybe does something else.#2021-03-1909:53danielnealaha ok thanks 🙂#2021-03-1910:05ikitommithis should also work: [:map [::some/ref {:optional true}]], no need to repeat the schema for refs.#2021-03-1910:21danielnealnice!#2021-03-1910:59danielnealIf I'm walking a schema and want to convert all schemas of type ::sq/ref into :string, but I also want to deref all other refs, how do I do this/ This is what I've got so far:
(defn output-schema
  [schema]
  (malli/walk
   schema
   (malli/schema-walker
    (fn [schema]
      (malli/type schema)
      (cond
        (= schema ::sq/ref) :string
        (= (malli/type schema) :map)
        .... do some other stuff
        :else schema)))
   {::malli/walk-schema-refs true
    ::malli/walk-refs true}))
But the schema is expanded by the time the check happens, so it fails
#2021-03-1911:00ikitommiwhat does (malli/walk schema (malli/schema-walker identity) {::malli/walk-schema-refs true, ::malli/walk-refs true}) do?#2021-03-1911:11danielnealyep, that expands all#2021-03-1911:01ikitommithat might expand all automatically#2021-03-1911:09ikitommi
(m/walk
  [:schema
   {:registry {"Country" [:map
                          [:name [:enum :FI :PO]]
                          [:neighbors [:vector [:ref "Country"]]]]
               "Burger" [:map
                         [:name string?]
                         [:description {:optional true} string?]
                         [:origin [:maybe "Country"]]
                         [:price pos-int?]]
               "OrderLine" [:map
                            [:burger "Burger"]
                            [:amount int?]]
               "Order" [:map
                        [:lines [:vector "OrderLine"]]
                        [:delivery [:map
                                    [:delivered boolean?]
                                    [:address [:map
                                               [:street string?]
                                               [:zip int?]
                                               [:country "Country"]]]]]]}}
   "Order"]
  (m/schema-walker #(mu/update-properties % assoc :type (m/type %)))
  {::m/walk-schema-refs true})
;[:schema
; {:registry {"Country" [:map [:name [:enum :FI :PO]] [:neighbors [:vector [:ref "Country"]]]],
;             "Burger" [:map
;                       [:name string?]
;                       [:description {:optional true} string?]
;                       [:origin [:maybe "Country"]]
;                       [:price pos-int?]],
;             "OrderLine" [:map [:burger "Burger"] [:amount int?]],
;             "Order" [:map
;                      [:lines [:vector "OrderLine"]]
;                      [:delivery
;                       [:map [:delivered boolean?] [:address [:map [:street string?] [:zip int?] [:country "Country"]]]]]]},
;  :type :schema}
; [:malli.core/schema
;  {:type :malli.core/schema}
;  [:map
;   {:type :map}
;   [:lines
;    [:vector
;     {:type :vector}
;     [:malli.core/schema
;      {:type :malli.core/schema}
;      [:map
;       {:type :map}
;       [:burger
;        [:malli.core/schema
;         {:type :malli.core/schema}
;         [:map
;          {:type :map}
;          [:name [string? {:type string?}]]
;          [:description {:optional true} [string? {:type string?}]]
;          [:origin
;           [:maybe
;            {:type :maybe}
;            [:malli.core/schema
;             {:type :malli.core/schema}
;             [:map
;              {:type :map}
;              [:name [:enum {:type :enum} :FI :PO]]
;              [:neighbors [:vector {:type :vector} [:ref {:type :ref} "Country"]]]]]]]
;          [:price [pos-int? {:type pos-int?}]]]]]
;       [:amount [int? {:type int?}]]]]]]
;   [:delivery
;    [:map
;     {:type :map}
;     [:delivered [boolean? {:type boolean?}]]
;     [:address
;      [:map
;       {:type :map}
;       [:street [string? {:type string?}]]
;       [:zip [int? {:type int?}]]
;       [:country
;        [:malli.core/schema
;         {:type :malli.core/schema}
;         [:map
;          {:type :map}
;          [:name [:enum {:type :enum} :FI :PO]]
;          [:neighbors [:vector {:type :vector} [:ref {:type :ref} "Country"]]]]]]]]]]]]]
#2021-03-1911:10ikitomminot sure if that’s near what you want to do.#2021-03-1911:13danielnealAh I think I said :type wrong, the type is :malli.core/schema, what I'm checking for is the schema itself to be ::sq/ref#2021-03-1911:14danielnealSo is what you're suggesting to do two walks, one to capture information and put it in the properties, and then a second to do the other transformations?#2021-03-1911:18danielnealhmm#2021-03-1911:20danielnealI suppose I could put some property on the ::sq/ref schema itself, but it feels like I'm missing a trick#2021-03-1911:28nilernTo do it in one traversal you would need a more Visitor-like prewalk where you manually deref the ones you want. I think that can be done with -walk and Walker but only schema-walker and walk are documented and stable ATM#2021-03-1911:31danielnealyeah that makes sense, you kind of prewalk it and choose whether to deref or swap as you descend#2021-03-1912:38ikitommiok, I guess I missed the original point. But there are public walkers like https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L37-L51#2021-03-1915:00danielnealis it safe to use those functions and protocols?#2021-03-1915:03nilernhttps://github.com/metosin/malli/#alpha#2021-03-1915:03nilern> extender api: public vars, name starts with -, e.g. malli.core/-collection-schema. Not needed with basic use cases, might evolve during the alpha, follow https://github.com/metosin/malli/blob/master/CHANGELOG.md for details#2021-03-1915:27danielnealnice, i see#2021-03-1911:31danielnealyeah that makes sense, you kind of prewalk it and choose whether to deref or swap as you descend#2021-03-1911:32danielnealthanks#2021-03-1917:36raymcdermottdoing the graphviz thing is a bit whack it turns out cos it passes through all of the constraints to be visualised and it soon gets ugly#2021-03-1917:36raymcdermott#2021-03-1917:38raymcdermottSo I guess you really have to pare back everything to the simplest of all possible models for nice visuals 🙂#2021-03-1917:42nilernMaybe we could have custom visualization props like we have :error/message?#2021-03-1918:45raymcdermottyes, or like {:swagger}#2021-03-1918:48raymcdermottif one had a Member, it would be nice for example to have {:viz/parent Org} on the map and {:viz/type string} on the name property#2021-03-1918:49raymcdermottand then one could select for those properties when transforming to DOT#2021-03-1921:54raymcdermott@ikitommi I might have been going about this wrong, so maybe this is now a better explanation of what I would like to have ...#2021-03-1921:55raymcdermott
(def Id [:re #"^[a-fA-F0-9]{8}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{4}-[a-fA-F0-9]{12}$"])
(def Name [:string {:min 3 :max 255}])

(def org-ref :ref)
(def org-ref-viz [:map [org-ref :string]])
(def org-ref-description "Reference (name or ID) of the organisation")
(def org-ref-valid [:map {:title org-ref-description} [org-ref Name]])
(def org-ref-swagger {:swagger {:description org-ref-description
                                :example     "Acme tech, Houston TX"}})
(def Org-Ref [:map {:title (name org-ref)}
              [org-ref org-ref-swagger Name]])

(def org-id :id)
(def org-id-viz [:map [org-id :string]])
(def org-id-description "The organisation ID")
(def org-id-valid [:map [org-id Id]])
(def org-id-swagger {:swagger {:description org-id-description
                               :example     (->id)}})

(def Org-Id [:map {:title (name org-id)}
             [org-id org-id-swagger Id]])

(def org-viz {"Org" (mu/merge org-id-viz org-ref-viz)})
(def Org (mu/merge Org-Id Org-Ref))
#2021-03-1921:56raymcdermottI can't find a way to create org-ref-full by combining org-ref-valid with org-ref-swagger#2021-03-1921:57raymcdermottI have added org-ref-viz as that's what I would like to derive from org-ref-valid#2021-03-1921:59raymcdermottAt the moment I am OK with doing it the way as it is above cos there is still a lot of value but obviously there seems to be quite a bit of boiler-plate#2021-03-2111:02Yevgeni TsodikovHey, This might be a silly question. Is it possible to validate a map while ignoring optional fields if they are invalid? for example it would return false here:
(m/validate [:map 
             [:a string?]
             [:b {:optional true} int?]]
            {:a "Hey" :b "Nope"})
=> false
But there would be a way to conform a map with invalid optional fields and “sanitize” the map
; This made-up function would return {:a "Hey"}
(m/conform [:map 
            [:a string?] 
            [:b {:optional true} int?]]
           {:a "Hey" :b "Nope"})
#2021-03-2111:15ikitommiDo you mean the first should return true?#2021-03-2112:00HankstenbergHm, shouldn't a validation with an open map that just containts the schema for :a do the trick? Or let b be any? Under what circumstances should b be an int?#2021-03-2112:45ikitommi> Is it possible to validate a map while ignoring optional fields if they are invalid? for that, yes, you can transform the schema before validation, just convert optional values to :any. but if you would like to strip away invalid values, you could: 1. run m/explain on data 2. recursively remove all values in :errors :in path. the invalid paths are part of the explain result:
(m/explain
  [:map
   [:a string?]
   [:b {:optional true} int?]]
  {:a "Hey" :b "Nope"})
;{:schema [:map [:a string?] [:b {:optional true} int?]],
; :value {:a "Hey", :b "Nope"},
; :errors (#Error{:path [:b], :in [:b], :schema int?, :value "Nope"})}
#2021-03-2113:11Hankstenberg@U055NJ5CC That's something I'd like to be able to do too. Is there an idiomatic way to do that?#2021-03-2113:15Yevgeni Tsodikov> Do you mean the first should return `true`? Yes 🙂 > recursively remove all values in `:errors` `:in` path. the invalid paths are part of the explain result: The :optional data is missing from the :schema field, how can I know which field is safe to remove and which field makes the map actually invalid?#2021-03-2113:19Yevgeni TsodikovMy scenario is an API with some optional fields. If the client doesn’t send then -> no worries. If the client sent some invalid optional fields, in some cases I’d like to pass the request and not fail it. (Such cases may be that the client is a mobile device with a crappy sdk, which might report invalid data for optional fields)#2021-03-2113:34Yevgeni TsodikovAlso - > Do you mean the first should return `true`? Not without some hints or options.
; This is correct
(m/validate [:map 
             [:a string?]
             [:b {:optional true} int?]]
            {:a "Hey" :b "Nope"})
=> false

; Additional options
(m/validate [:map 
             [:a string?]
             [:b {:optional true} int?]]
            {:a "Hey" :b "Nope"}
            {:fail-on-optional? false)
=> true
#2021-03-2112:45ikitommi> Is it possible to validate a map while ignoring optional fields if they are invalid? for that, yes, you can transform the schema before validation, just convert optional values to :any. but if you would like to strip away invalid values, you could: 1. run m/explain on data 2. recursively remove all values in :errors :in path. the invalid paths are part of the explain result:
(m/explain
  [:map
   [:a string?]
   [:b {:optional true} int?]]
  {:a "Hey" :b "Nope"})
;{:schema [:map [:a string?] [:b {:optional true} int?]],
; :value {:a "Hey", :b "Nope"},
; :errors (#Error{:path [:b], :in [:b], :schema int?, :value "Nope"})}
#2021-03-2114:06ikitommi#2021-03-2114:07ikitommiping @U01PU4APSKV#2021-03-2114:22Yevgeni TsodikovThat’s great, thanks @U055NJ5CC!#2021-03-2114:35Yevgeni TsodikovIs there a way to conform/remove the invalid optional fields?#2021-03-2114:51ikitommiyes. Many ways to do that : 1) attach a custom transformer to the new :any fields to strip those away 2) in case of error, call m/explain and remove all values from paths that have error#2021-03-2114:51ikitommidon't have time to write an example, but definitely doable 😉#2021-03-2116:18Yevgeni TsodikovWhat’s your opinion on something like:
(defn allow-invalid-optional-values [schema]
  (m/walk
    schema
    (m/schema-walker
      (fn [s]
        (cond-> s
                (m/entries s)
                (mu/transform-entries
                  (partial map (fn [[k {:keys [optional] :as p} s]]
                                 (if optional
                                   [k
                                    (assoc p :original-spec s
                                             :decode/remove-invalid-fields {:compile (fn [schema _]
                                                                                       (fn [x]
                                                                                         (if-let [original-spec (:original-spec (m/properties schema))]
                                                                                           (when (m/validate original-spec x)
                                                                                             x)
                                                                                           x)))})
                                    :any]
                                   [k p s])))
                  (m/options s)))))))

(-> [:map
     [:a string?]
     [:b {:optional true} int?]]
    allow-invalid-optional-values
    (m/decode
      {:a "Hey" :b "Nope"}
      (mt/transformer {:name :remove-invalid-fields})))
=> {:a "Hey", :b nil}
I don’t like having multiple transformers, though. How can I unify them? Similarly to what malli offers with mt/strip-extra-keys-transformer ?
#2021-03-2114:12ikitommias the slack history rolls out fast, pasted the example into https://github.com/metosin/malli/blob/master/docs/tips.md#allowing-invalid-values-on-optional-keys#2021-03-2115:37Yevgeni TsodikovThe allow-invalid-optional-values function is missing the (m/options s) arg of mu/transform-entries#2021-03-2116:31ikitommihttps://github.com/metosin/malli/commit/da6366dd845b300923a5aec28eae96f15a8c9f71#2021-03-2116:15respatializedhi! I've been really impressed with malli so far. really appreciate all the work you've put into it. I'm trying to generate sample values from a schema, and I'm encountering a case where :orn fails and :altn succeeds:
(mg/generate [:altn [:bool boolean?] [:num int?]] {:seed 20})
=> [true]
(mg/generate [:orn [:bool boolean?] [:num int?]] {:seed 20})
=> Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:16).
:malli.generator/no-generator {:schema [:orn [:bool boolean?] [:num int?]], :options {:seed 20}}
Is this expected at this stage? Is there additional implementation for :orn generators that still needs to be done, or is this a bug?
#2021-03-2116:17respatializedsimple :or also succeeds on this case:
(mg/generate [:or boolean?  int?] {:seed 20})
=> true
#2021-03-2116:49ikitommi@afoltzm 🙇 , see https://github.com/metosin/malli/pull/400#2021-03-2116:54ikitommimerged in master#2021-03-2117:04respatializedwow, that was fast! thanks!#2021-03-2117:05borkdudeit's weekend, the time where magic OSS happens#2021-03-2118:09ikitommipushed out [metosin/malli "0.3.1"], finally with a working cljdoc - https://cljdoc.org/d/metosin/malli/0.3.1/doc/changelog.#2021-03-2207:00robert-stuttafordhaha just my luck, a week after i get started, the docs improve 😆 it looks great!#2021-03-2207:01robert-stuttafordbtw @ikitommi i ended up chunking the results of sets of 100 item m/provide into an atom and then m/union ing them together afterward. i ended up with a fair bit of [:or [:or [:or ....]]] but that was easy to remedy with a postwalk 🙂#2021-03-2207:33ikitommi@robert-stuttaford good to hear. the right solution is to add few new Protocols (e.g. Inferrer and Provider) and attach the latter to the reified IntoSchemas: each instance know how to construct itself given a value and options. Internally using type/class-based lookup tables, it should be 2-4 orders of magnitude faster, and pluggable. Have an pre-initial draft somewhere 🙂#2021-03-2207:57robert-stuttafordthat sounds sane!#2021-03-2210:06Panel
(malli/encode
  [:or [:map
           [:password string?]
           [:password2 string?]]
     [:fn {:error/message "passwords don't match"}
      '(fn [{:keys [password password2]}]
         (= password password2))]]
  {:password "ok"}
  (mt/key-transformer {:encode  name}))
=> {:password "ok"}
(malli/encode
  [:and [:map
           [:password string?]
           [:password2 string?]]
     [:fn {:error/message "passwords don't match"}
      '(fn [{:keys [password password2]}]
         (= password password2))]]
  {:password "ok"}
  (mt/key-transformer {:encode  name}))
=> {"password" "ok"} Why doesn't encode work on the first one but does on the second, related to using :or instead of :and I guess.
#2021-03-2210:11ikitommiI believe neither of the examples doesn’t do anything, they are just just no-op.#2021-03-2210:12ikitommioh, wait.#2021-03-2210:14nilernencode does not validate and those inputs are invalid, so nasal demons#2021-03-2210:15ikitommiactually:
(-transformer [this transformer method options]
            (let [this-transformer (-value-transformer transformer this method options)]
              (if (seq children)
                (let [transformers (mapv #(or (-transformer % transformer method options) identity) children)
                      validators (mapv -validator children)]
                  (-intercepting this-transformer
                                 (if (= :decode method)
                                   (fn [x]
                                     (reduce-kv
                                       (fn [x i transformer]
                                         (let [x* (transformer x)]
                                           (if ((nth validators i) x*) (reduced x*) x)))
                                       x transformers))
                                   (fn [x]
                                     (reduce-kv
                                       (fn [x i validator] (if (validator x) (reduced ((nth transformers i) x)) x))
                                       x validators)))))
                (-intercepting this-transformer))))
#2021-03-2210:15nilernAlthough in terms of the implementation I think both of those should be no-ops#2021-03-2210:15ikitommiit does validate and the first one fails -> doesn’t transform#2021-03-2210:18nilernAh yes so :or and seqex schemas validate when encoding because they have to pick a branch but nothing else (e.g. :and does) But that is implementation detail#2021-03-2210:23nilernThe recommended flow is first validate, then encode if ok, else explain Ceterum censeo, maybe some day we will have a thing that does all of that in one pass like Plumatic coerce#2021-03-2210:25PanelThanks for the explanations !#2021-03-2210:45nilernThis is my pet peeve and now I finally added an issue https://github.com/metosin/malli/issues/404 But it is a big decision and feature#2021-03-2214:18robert-stuttafordis it possible for malli to report all errors instead of stopping on the first one?#2021-03-2214:19nilernexplain(er) should report all the errors. What are you seeing / not seeing?#2021-03-2214:20robert-stuttafordah, apologies. i'm using explain -> with-spell-checking -> humanize#2021-03-2214:21robert-stuttafordseems humanize is dropping it#2021-03-2214:21ikitommireally? should not.#2021-03-2214:23robert-stuttafordlemme make reaaaal sure first, don't want to waste your time 😅#2021-03-2214:29robert-stuttafordok, it's not humanize. in the explain, the last error that i get is a :malli.core/input-remaining . when i fix that issue, then i get a new previously unreported error further down (happens to be the same issue: a kv-pair that should be one map level up)#2021-03-2214:30robert-stuttafordso, is there some way have the whole sequence reported on at once?#2021-03-2214:31robert-stuttafordbasically, i'd like to use malli specs to provide a structural EDN linter in a web-based cms code editor. but for this one thing, it's working, beautifully#2021-03-2214:33robert-stuttafordi am getting other errors from other collections, but the same issue manifests, in that i only get one error per such collection. i guess the question i'm asking is, how do i make :sequential and/or :+ check all elements#2021-03-2214:34ikitommiOh, the partial seqexp, no reporting / transformation on partially matched sequences atm. https://github.com/metosin/malli/issues/387#2021-03-2214:34robert-stuttafordok so this is because i'm using :+ rather than :sequential?#2021-03-2214:35ikitomminot sure, could you give an example with actual and expected result?#2021-03-2214:36robert-stuttafordok, i switched to sequential, and this is the reason - thank you!#2021-03-2214:36nilernYou should use e.g. :sequential instead of :*/`:+` if you can#2021-03-2214:36robert-stuttafordi can live without ensuring the sequence has at least one item#2021-03-2214:37nilern[:sequential {:min 1} :int]#2021-03-2214:40robert-stuttafordok rad i've got what i need. malli is lovely#2021-03-2214:41robert-stuttafordnow to use edamame's edn metadata stuff to wire up go-to-line links#2021-03-2214:42nilernIn general the seqex engine can't "resynchronize" after an erroneous element. We could have some optimizer that recognizes [:+ :int] and transforms it to [:sequential {:min 1} :int] but it would add complexity and unpredictability...#2021-03-2214:42robert-stuttafordtotally fair#2021-03-2214:45ikitommidid a malli + edamame poc some time ago, can’t recall was it any good, but pushed just as a gist: https://gist.github.com/ikitommi/0e5c4e48d8aeb7dd176128856ecdacb5#2021-03-2214:45ikitommisomething @borkdude was asking I think.#2021-03-2214:47borkdudewhat I tried to do there is "complect" error message with locations directly#2021-03-2214:48borkdudebut I think you can just do explain and then look up the vals in the paths which have locations (or maybe they are already in the errors) and then postwalk the result to add the locations to the error messages#2021-03-2214:50robert-stuttafordthat second thing is what i'm doing#2021-03-2214:56robert-stuttafordyeah @ikitommi your gist is the beautiful general version of the ugly hack i'm busy writing. going to start again with yours, thank you#2021-03-2214:57nilernThat code looks familiar :thinking_face:#2021-03-2214:58borkdude@robert-stuttaford I think the second thing is probably the easiest?#2021-03-2215:02ikitommilooking at the code, it has transduce, I never use that. I can’t recall the origins of the code, just from one of the zillion scratch files. could be @nilern s work too :thinking_face:#2021-03-2215:06nilernLooking at the Metosin Slack history I did the unwrap and collect bits and remarked that the prewalk makes it O(n^2) but it only matters if there are large map keys or set members which hardly ever happens...#2021-03-2215:08nilernAnyway it took a while to write so it would be cool if it found some use#2021-03-2306:21robert-stuttafordthanks all, i'll report back on how it goes!#2021-03-2308:41ikitommi@raymcdermott I can look your thing today/tomorrow.#2021-03-2311:33ikitommi@d.ian.b no inbuilt mapping for fn?. But, you can:
(m/validate [:fn fn?] (partial +))
; => true
#2021-03-2311:34Ian Fernandezyeah#2021-03-2311:34ikitommialso:
(m/explain
  [:=> [:cat [:* [:int {:gen/min -1000, :gen/max 1000}]]] :int] 
  (partial +)
  {::m/function-checker mg/function-checker})
; => nil
#2021-03-2311:34Ian FernandezI've made it right now haha#2021-03-2311:34ikitommi
(m/explain
  [:=> [:cat [:* [:int {:gen/min -1000, :gen/max 1000}]]] [:int {:min 1}]]
  (partial +)
  {::m/function-checker mg/function-checker})
;{:schema [:=> [:cat [:* [:int #:gen{:min -1000, :max 1000}]]] [:int {:min 1}]],
; :value #object[clojure.core$_PLUS_ 0x6edca2b1 "
#2021-03-2311:35Ian Fernandeznice!#2021-03-2311:57Ian FernandezHow I can use a register of the (m/function-schemas) registry, onto m/validate?#2021-03-2316:06Ian Fernandez(get-in (m/function-schemas) [(quote *ns*) 'foo :schema])#2021-03-2316:29ikitommi@d.ian.b good question, atm, no easy way. Idea with the function registry is that there will be a instrument kinda thing, that will wrap a) some b) all registered function schmaas like Orchestra - running input & output validation. Could be also used to emit generated data based only on the function definitions. Ideas and PRs welcome on that.#2021-03-2316:30ikitommialso, did a spike on infferring schemas from normal vars. you get useful guesses pretty easily, better with tools.analyzer & clj-kondo and I guess. really good with core.typed.#2021-03-2316:36ikitommithe new kw-varargs thing would work nicely with global registry, my guess is that the core team will plug into that with spec. e.g. given a function:
(defn doit [& {:domain.user/keys [id name]}] [id name])
… running (m/collect #'doit) would infer a [:map :domain.user/id :domain.user/name] out of it, register it as a malli function schma, after (m/instrument my-registry) saying:
(doit :domain.user/id 1, :domain.user/name "kikka")
would cause it to run validation.
#2021-03-2316:38ikitommigiven there is few hours extra time, I would write a sample code so that I could prove a point that it’s doable and awesome.#2021-03-2316:45raymcdermottI don't seem to be able to merge maps with [:and] constraints#2021-03-2316:47raymcdermott
(def x [:and [:map
              [:start int?]
              [:end int?]]
        [:fn (fn [{:keys [start end]}]
               (< start end))]])
=> #'user/x
(def y [:map
        [:here int?]
        [:there int?]])
=> #'user/y
(require '[malli.util :as mu])
=> nil
(mu/merge y x)
=> [:and [:map [:start int?] [:end int?]] [:fn #object[user$fn__4910 0x40fa91ef "
#2021-03-2316:47raymcdermottis this expected?#2021-03-2316:54borkdudeand should merge combine the predicate in another predicate which ands those if both maps have predicates?#2021-03-2316:55raymcdermottit should invoke both functions ... maybe the order would not be predictable but I'll take that#2021-03-2316:55borkdudeprobably the order of merge args#2021-03-2316:56borkdudethis could lead to funny problems, like what if you merge in a closed map in an open map, probably the resulting map should be open?#2021-03-2316:59borkdudeor should merge consider predicates and other properties like metadata, which is ignored in merge args?#2021-03-2317:00borkdude(with-properties-of (merge x y) y)#2021-03-2317:54raymcdermottmerge takes the last as the 'winner' so I think that would be the most idiomatic#2021-03-2317:54raymcdermottbut I agree that merging things that are not maps is tricky#2021-03-2319:19ikitommiPlumatic dropped s/both in favour of s/constrained just because the first is not a good idea: “apple and fruit and a car” please. Currently :and already kinda means “the first thing constrained with the rest” as we pick the generator from first and then constraint with the rest using gen/such-that. Given that, we could make :and mergable, would merge with the first and keep the rest as extra leaves of :and? e.g.
[:map ::x]
[:map ::y] 
; => [:map ::x ::y]

[:map ::x]
[:and [:map ::y] map?] 
; => [:and [:map ::x ::y] map?]

[:and [:map ::x] map?]
[:map ::y] 
; => [:and [:map ::x ::y] map?]

[:and [:map ::x] map?]
[:and [:map ::y] map?] 
; => [:and [:map ::x ::y] map? map?]

[:and [:map ::x]]
map? 
; => map?
would that be … more correct?
#2021-03-2319:21ikitommihaving [:and [:map …] [:fn …]] is quite common, having it non-mergable is a bummer.#2021-03-2421:19raymcdermottif both maps have [:and [:map ..][fn...] it could be rejected. If one has an [:and ...] and the other doesn't can't you still merge the maps? I am probably under thinking it 🙂#2021-03-2322:07PanelAnyway to have a dynamic default ? Like if I want a default timestamp to be generated when runing decode with default-value-transformer ?#2021-03-2323:17Panel
(malli/decode
 [:map {:registry
        {:inst (m/-simple-schema
                {:type :inst
                 :pred inst?})}}
  [:time :inst]
  [:id1 :uuid]
  [:id2 :uuid]]
 {}
 (mt/default-value-transformer
  {:defaults {:inst (constantly (rand-int 100))
              :uuid (constantly (char (rand-int 100)))}}))
Found something that works for me, but the default are generated once per type so for example if you need two different uuid it won't work.
#2021-03-2323:34Panel
(mu/optional-keys
 [:map {:registry
        {:inst (m/-simple-schema
                {:type :inst
                 :pred inst?})}}
  [:time :inst]
  [:id :uuid]])
This throw a java.lang.StackOverflowError
#2021-03-2401:56PanelIt does work if the registry is passed in the options map
(mu/optional-keys
 [:map
  [:time :inst]
  [:id :uuid]]
 {:registry
  (mr/composite-registry
   m/default-registry
   {:inst (m/-simple-schema
           {:type :inst
            :pred inst?})})})
#2021-03-2413:16respatializedIs there a preferred way to ensure that a seqex schema is a given collection type? right now I can use [:and vector? [:catn [:bool :boolean] [:int :int]]], but I was wondering if there was a more canonical or performant way of doing it (generators often fail to satisfy the predicate, for example).#2021-03-2413:23nilernNo. I think it only matters for generators and :and is in general not generator-friendly (currently the generator is a gen/such-that, which is pretty bad but how much better can we even get?)#2021-03-2413:25nilern(For that particular example I would probably use [:tuple :boolean :int] but you might really have something that actually needs seqex)#2021-03-2413:29respatializedyeah, that was a deliberately simplified example (the actual motivating use case is validation on hiccup forms). I think I'll need to take a crack at custom generators for this use case.#2021-03-2413:33nilernI think I've seen some discussion of this previously and it should be simple to add. So maybe file an issue as well?#2021-03-2413:51nilernIt is unfortunate that we have to add non-orthogonal props like [:string {:min 5}] and now maybe [:catn {:type :vector}] to get decent generators#2021-03-2414:04respatializedminimallist has an additional https://github.com/green-coder/minimallist/blob/f10ebbd3c2b93e7579295618a7ed1e870c489bc4/src/minimallist/helper.cljc#L72 that does this, minimallist.helper/in-vector, but it chooses a map-based representation for schemas that allows additional metadata of :coll-type to be assoc'd to the schema. Unclear how something similar would work in malli.#2021-03-2414:08nilernIt would just add that to props but the seqex schemas still have to be updated to use that information#2021-03-2518:38ikitommifirst kinda BREAKING change in the public api. m/parse on :multi now returns the branch information. It’s a fix, but still:
(def Multi
  [:multi {:dispatch :type}
   [:user [:map [:size :int]]]
   [::m/default :any]])

(m/parse Multi {:type :user, :size 1})
; => [:user {:type :user, :size 1}]

(m/parse Multi {:type "sized", :size 1})
; => [:malli.core/default {:type "sized", :size 1}]

(->> {:type :user, :size 1}
     (m/parse Multi)
     (m/unparse Multi))
; => {:type :user, :size 1}
#2021-03-2522:21ikitommi+5 loc later (including comments): :and merges on first child! comments welcome on the results - https://github.com/metosin/malli/pull/405#2021-03-2608:11ikitommiping @U04V5V0V4#2021-03-2616:20raymcdermottI thought it read 5k loc ... so 😅#2021-03-2616:21raymcdermottI have added a quick comment. Thanks for the feature#2021-03-2907:14Yevgeni TsodikovAwesome MR! Thanks 🙏#2021-03-2910:34ikitommimerged in master#2021-03-2621:15borkdudeI'm in the process of designing a task-executor DSL. Example: {:tasks {:clean [:shell "rm" "-rf" "target"]}} I want to be able to give a task a description and possibly more options. So I thought I can do: {:tasks {:clean [:shell {:task/description "foo"} "rm" "-rf" "target"]}} but this reads a bit weird, because I expect :shell to get shell-related options and not some general task options. One alternative I came up with:
{:tasks {:clean ^{:task/description "foo"} [:shell "rm" "-rf" "target"]}}
A bit ugly maybe, but semantically correct: the metadata is about the task. However, processing EDN with metadata, it can be done, but it might lead to confusion. The other alternative, an explicit :task wrapper if you want to provide options:
{:tasks {:clean [:task {:description "foo"} [:shell "rm" "-rf" "target"]]}}
I wonder if you came across similar problems when designing malli, I'd love to hear your feedback.
#2021-03-2621:26borkdudeI think malli more or less ended up with:
{:tasks {:clean [:shell {:description "Clean all the things"} "rm" "-rf" "target"]}}
right?
#2021-03-2710:29borkdudeI think I'll land on:
{:tasks {:clean [:shell {:description "foo"} "rm -rf target"]}}
#2021-03-2710:29borkdudethanks duckie :)#2021-03-2711:36PanelJust curious if you are using Malli to make the DSL ?#2021-03-2711:37borkdudenot really no#2021-03-2711:42borkdudealthough malli would probably be a good fit for validation#2021-03-2713:34borkdudeLanded on this one:
{:tasks {:clean {:description "Foo" :task [:shell "rm -rf target"]}}}
#2021-03-2713:35borkdudeNot hiccup as usual, but I think the cleanest so far#2021-03-2714:17ikitommi@borkdude looks good, I would have suggested a map too, keeps the concepts separated.#2021-03-2714:18ikitommiMalli also has the generic map-syntax and have explored a third, simpler (clj-fx-like) map-syntax.#2021-03-2714:23ikitommiNeed to pick a "better" map-syntax for malli 1.0.0, but will be first-class support, in malli.core. One can't mix syntaxes (terse/hiccup and map) freely, just one allowed per AST fragment.#2021-03-2714:56borkdude@ikitommi what is clj-fx-like map syntax? would love to see it#2021-03-2904:58ikitommi@borkdude wrote an issue of the syntax, with examples: https://github.com/metosin/malli/issues/406#2021-03-3007:22Adam HelinsIs there a way of generating sorted maps using :map-of ? Spec allows that and it is indeed very useful when working with sorted colls.#2021-03-3014:32nilernI don't think there is any support for sorted colls specifically#2021-03-3019:24Adam HelinsInteresting, i can't find anything, no issue, nothing in the codebase...#2021-04-0108:58nilernIt never comes up in input validation and our DbC story is not really there yet. Whereas Spec was basically created for DbC and third party tools are needed for most other things.#2021-03-3013:12Adam HelinsHmmm, took me a while to realise that was a source of error and I cannot seem to find how to merge such maps:
[:merge
 [:map [:a :int]]
 [:and
  [:map [:b int]]
  [:fn some-pred]]]
The problem is that the second map is enclosed by [:and] , and Malli doesn't seem to understand that it is ultimately a map, meaning that :merge does not behave like it is merging 2 maps. How can I merge them while keeping that custom predicate?
#2021-03-3014:30nilernhttps://github.com/metosin/malli/pull/405 (but still unreleased)#2021-03-3019:20Adam HelinsExcellent, once again @U055NJ5CC saves the day#2021-03-3113:46ikitommishipped in 0.4.0 now#2021-03-3105:23valeraukoAfter seeing the #Performance section of malli's readme I'm quite interested in trying it out. One thing I was wondering is how it fares at compile-time? I'm just assuming that it achieves high runtime speed by offloading as much as possible to macros. Are there any numbers how this affects eg lein startup time?#2021-03-3105:29ikitommi@vale good question. There are no macros in malli (ok, one mandatory). the “schema compile time” is at runtime, you can have a separate step to create an immutable & optimized validator/transformer/parser and reuse that.#2021-03-3105:30ikitommiin many cases, even without reuse, it’s still faster than with alternatives, e.g. with spec / spec-tools - one-shot “gimme a validator, use it once and forget it”.#2021-03-3108:11valeraukoawesome, time to dive in then! thanks for the great work!#2021-03-3105:32ikitommimalli.core (without sci) should load under 1sec from source and 0.1sec from classes.#2021-03-3105:34ikitommi
(= int? (m/validator [:and int? int? [:and int? [:and int?]]]))
; => true
#2021-03-3105:46ikitommi
(= identity
   (m/decoder
     [:map
      [:id :int]
      [:name :string]
      [:tags [:vector :string]]]
     (mt/json-transformer)))
; => true
#2021-03-3107:24robert-stuttafordwas hoping that humanize would use language like 'at least one string? is required' here, but instead, i get "unknown error". am i doing it wrong, or is this a TODO for humanize?
(m/explain [:sequential {:min 1} string?] ())

  ;; {:schema [:sequential {:min 1} string?]
  ;;  :value  ()
  ;;  :errors (#Error{:path   []
  ;;                  :in     []
  ;;                  :schema [:sequential {:min 1} string?]
  ;;                  :value  ()
  ;;                  :type   :malli.core/limits})}

  (-> (m/explain [:sequential {:min 1} string?] ())
      me/humanize)

  ;; ["unknown error"]
#2021-03-3107:46ikitommi@robert-stuttaford just missing an error mapping for :malli.core/limits. PR welcome to malli.error.#2021-03-3107:46ikitommiyou can also play with the mappings by passing them into me/humanize:
(-> (m/explain [:sequential {:min 1} string?] ())
    (me/humanize {:errors {::m/limits {:error/message {:en "there's no limits"}}}}))
; => ["there's no limits"]
#2021-03-3107:52ikitommiuncomplete, but close:
(defn min-max-sequence-error-message [error options]
  (str ((me/-pred-min-max-error-fn {:pred identity}) error options) " elements"))

(-> (m/explain [:sequential {:min 1} string?] ())
    (me/humanize {:errors {::m/limits {:error/fn {:en min-max-sequence-error-message}}}}))
; => ["should be at least 1 elements"]
#2021-03-3108:05Adam HelinsThe following scheme systematically produces a stack overflow. It is a recursive definition where one item (`:E`) contains another item (which might be :E as well, or not). I am surprised this stack overflow is that systematic. There must be something wrong somewhere unless I am missing the obvious. Tried with :or instead of :multi, same thing.
(def registry
     {:A    [:tuple [:= :A]]
      :B    [:tuple [:= :B]]
      :C    [:tuple [:= :C]]
      :D    [:tuple [:= :D]]
      :E    [:tuple
             [:= :E]
             :item]
      :item [:multi
             {:dispatch first}
             [:A :A]
             [:B :B]
             [:C :C]
             [:D :D]
             [:E :E]]})


(malli.gen/generate [:schema
                     {:registry registry}
                     :item])
#2021-04-0108:33nilernThere was https://github.com/metosin/malli/issues/359 also#2021-03-3108:07Adam Helins
Execution error (StackOverflowError) at malli.core/-schema (core.cljc:242).
#2021-03-3108:11Adam HelinsValidation results in the same, I probably should open an issue actually#2021-03-3108:11valeraukoawesome, time to dive in then! thanks for the great work!#2021-03-3108:32ikitommi@adam678 would [:ref :item] work? It's a lazy reference type#2021-03-3108:40Adam HelinsIn the :E definition? Such as:
[:tuple
 [:= :E]
 [:ref :item]]
I've tried it different ways but get an exception: :malli.core/invalid-ref {:ref :item} Sorry, seems I didn't drink enough coffee today 😅
#2021-03-3108:41Adam HelinsRight, refs must be namespaced...#2021-03-3109:10Adam Helins@ikitommi However, there is still something wrong with generation. Adding another recursive item makes things very slow (eg. 2-3 seconds for generating something like [:E [:E [:A]]]. Adding a third recursive item makes things so slow that my fans are churning and after a couple of minutes there still isn't any result.#2021-03-3109:26ikitommiplease write an issue, malli should lean on https://github.com/clojure/test.check/blob/master/doc/intro.md#recursive-generators, but it does not.#2021-03-3109:45Adam HelinsI did: https://github.com/metosin/malli/issues/408 I hope I am not spamming. It's just I don't quite remember having that sort of problems with Spec so I am having a bit of a hard time working with that. Not complaining though, so far using Malli has been very comfortable!#2021-04-0108:02ikitommihow about lazy generators? https://github.com/metosin/malli/pull/409#2021-03-3108:33ikitommithere is no recursion detection for eager references.#2021-03-3108:34ikitommiI Think StackOverflow is a correct way to fail here#2021-04-0108:35nilernIt makes sense but does leave something to be desired#2021-03-3108:58fugbixGreetings everyone!! Thank so much for this super neat lib, really cool and useful. I have a bit of an issue, whereby requiring malli.core triggers a ClassNotFoundException:
nREPL server started on port 55188 on host 127.0.0.1 - 
REPL-y 0.4.4, nREPL 0.8.3
Clojure 1.10.3
Java HotSpot(TM) 64-Bit Server VM 11.0.9+7-LTS
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
    Exit: Control+D or (exit) or (quit)
 Results: Stored in vars *1, *2, *3, an exception in *e

user=> (require '[malli.core :as m])
Execution error (ClassNotFoundException) at java.net.URLClassLoader/findClass (URLClassLoader.java:471).
malli.impl.util.SchemaError

user=>
my project is leiningen and includes many dependencies (including borkdude/sci because I am using multi dispatching, Datomic, Kafka etc). I am using the latest version of malli (`metosin/malli "0.3.1"`), Of course using malli from a toy lean leiningen project works perfectly fine. Although not a Clojure expert, I’ve never seen something similar. And I really can’t imagine why malli.impl.util doesn’t load? Any help would me much appreciated, I’ll keep digging!
#2021-04-0108:40nilernNo idea, I don't see anything nonstandard going on there#2021-04-0112:45fugbixThank you @U4MB6UKDL I have removed my target directory and everything’s fine now. Of course I would love to understand what brought it in this state, but I’ll carry on with work for now. Sorry for the fuss, and thanks again for that wicked malli lib, absolutely loving it.#2021-03-3109:08robert-stuttafordwonderful thank you @ikitommi!#2021-03-3109:26dharriganA little while ago, there was a conversation about stripping nulls from the return back to the client#2021-03-3109:26dharriganhjmmm...let me think a bit more...#2021-03-3109:42dharriganSo, okay, I'm doing this in the setup of the ring/router#2021-03-3109:42dharrigan
:muuntaja (m/create
                      (assoc-in m/default-options
                                [:formats "application/json" :opts]
                                {:mapper (-> (j/object-mapper)
                                             (.setSerializationInclusion JsonInclude$Include/NON_EMPTY))}))
#2021-03-3109:43dharriganFollowing throught the code, I believe I'm handed back an instance of the ObjectMapper from jsonista.core when I then set that option#2021-03-3109:43dharriganthe thing is, if I do this (now!), then all of my coercion fails (using malli)#2021-03-3109:43dharrigani.e.,#2021-03-3109:43dharrigan
"coercion": "malli",
    "errors": [
        {
            "in": [
                "repId"
            ],
            "message": "missing required key",
            "path": [
                "repId"
            ],
....
....
#2021-03-3109:44dharriganif I put it back to :muuntaja m/instance, then all is well with the world#2021-03-3109:44dharriganI'm a bit lost as to why it fails#2021-03-3109:45dharriganit feels like it's not able to parse the incoming JSON data from the request#2021-03-3110:37dharriganI realise this is in the wrong channel (maybe I should put into #reitit)#2021-03-3113:45ikitommipushed out [metosin/malli "0.4.0"]. It’s a new MINOR as the :multi parse is changed (fixed) and leaning on the old (broken) code can break someone. https://cljdoc.org/d/metosin/malli/0.4.0/doc/changelog#2021-04-0108:33nilernThere was https://github.com/metosin/malli/issues/359 also#2021-04-0108:19ikitommiWelcome lazy generators! recursive generators are now orders of magnitude faster, ping @adam678
(time
  (let [schema [:schema {:registry {::A [:tuple [:= :A]]
                                    ::B [:tuple [:= :B]]
                                    ::C [:tuple [:= :C]]
                                    ::D [:tuple [:= :D]]
                                    ::E [:tuple [:= :E] [:ref ::item]]
                                    ::F [:tuple [:= :F] [:ref ::item]]
                                    ::G [:tuple [:= :G] [:ref ::item]]
                                    ::item [:multi {:dispatch first}
                                            [:A ::A]
                                            [:B ::B]
                                            [:C ::C]
                                            [:D ::D]
                                            [:E ::E]
                                            [:F ::F]
                                            [:G ::G]]}}
                ::E]
        valid? (m/validator schema)]
    (is (every? valid? (mg/sample schema {:size 10000})))))
; => "Elapsed time: 230 msecs"
#2021-04-0108:27Adam HelinsThat was fast! Just tested it with my real world example and it seems to work just perfectly. (Generating WASM instructions where an instruction may contain a list of instructions). Thanks a lot!#2021-04-0207:44Adam HelinsI am wondering how Malli is performing in an inheritance-like scenario. Essentially, Malli allows keys to be :optional in :map . This is about having whole sets of keys being optional, in an all-or-none fashion, such as:
(def AnimalBase
     [:map ...])

(def Cat
     [:merge
      AnimalBase
      [:map ...]])

(def Dog
     [:merge
      AnimalBase
      [:map ...]])

(def Animal
     (m/schema [:multi {:dispatch :type-or-something} :cat Cat :dog Dog]))
Does it leverage the fact that the AnimalBase schema is mandatory in all dispatched schemas? Is there maybe an already existing better way of representing inheritance? Also assuming that being able to use generation is mandatory.
#2021-04-0208:09juhoteperi:merge with two map schemas creates a new map schema with keys from both schemas, so Cat and Dog don't refer to AnimalBase but are instead just map schemas with properties from the inputs#2021-04-0208:09juhoteperiOr hmph, I might be confusing it with malli.util/merge... not sure if this works similarly#2021-04-0208:12juhoteperi:merge should same as malli.util/merge#2021-04-0208:12juhoteperihttps://github.com/metosin/malli/blob/master/src/malli/util.cljc#L401#2021-04-0208:13Adam HelinsI believe malli.util/merge is indeed eager in that sense (creating a new map schema and forgetting about AnimalBase). But theoretically, since this example relies on data and :merge, that :multischema could deduce that all dispatched schemas are maps that could have a common set of keys. But it is just the most straightforward way of representing single inheritance IMO, maybe there is something better.#2021-04-0208:35Adam HelinsSorry I realized my example was a bit misleading so I deleted it so it doesn't spam the channel. I'll work more on this and try to come back with a real-world one.#2021-04-0310:55Setzer22Hi there! At our startup we're considering moving from spec to malli for our next project. I'm particularly excited about the clj-kondo integration support. Is there somewhere I could read more about it? Perhaps some of you using it? Did anyone integrate this into their repl/editor workflow (so that annotations are regenerated every time a new schema definition is made)? I'd also like to know how "deep" the static analysis goes. Is it just for elemental types like int or string? Or does the malli+clj-kondo combination understand your own compound types to some degree?#2021-04-0511:24ikitommihttps://twitter.com/puredanger/status/1378850899091578881#2021-04-0513:54emccueRelevant to the above, it feels like you could make spec out of malli but not the other way around#2021-04-0518:21nilernThere is more in the box for sure. On the other hand it's good that Spec does not have all that because it is in core#2021-04-0518:38emccueHow much of that "in the box"-ness is just about having a single maven artifact and how much is actual interdependency though?#2021-04-0519:00nilernAt least in theory we could just have the schema introspection and everything else could use that and live in separate jars#2021-04-0519:02nilernBut currently there is a monolithic schema protocol which does set a lower bound on the box size#2021-04-0519:05nilernSee also https://clojure.atlassian.net/browse/CLJ-2251 but I recall Tommi saying that walking Specs would still be janky#2021-04-0519:20marciolThere is spec/visitor in spec-tools @nilern. I think that the real problem is that it's impossible to decode serialized specs in runtime.#2021-04-0519:21marciolYou can go only one way without a proper way to reverse#2021-04-0519:29nilernYes the ticket mentions spec-tools and various details that I know little about since i never touched spec-tools#2021-04-0616:48ribeloIs it possible to create a key in a map using decode?#2021-04-0616:49ribelosomething like {:a 1 :b 2} => {:c 3}#2021-04-0617:27nilernSure it is#2021-04-0617:31nilern
(m/decode [:map {:decode/abc (fn [m]
                               (if (and (map? m) (contains? m :a) (contains? m :b))
                                 (-> m
                                     (dissoc :a :b)
                                     (assoc :c (+ (:a m) (:b m))))
                                 m))}
           [:a :int] [:b :int]]
          {:a 1, :b 2}
          (mt/transformer {:name :abc}))
=> {:c 3}
#2021-04-0617:34nilernhttps://github.com/metosin/malli#value-transformation does have this info although the explanations are a bit sparse#2021-04-0618:07ribeloI've read it, but somehow it didn't occur to me that I could use decode/encode on the whole map and not on an individual key#2021-04-0618:07ribelogreat!#2021-04-0618:09nilernYes that is the "key" to changing keys :drum_with_drumsticks:#2021-04-0618:43ikitommi👍 small nitpick: the example could be encode as it transforms the value out of schema. Or the schema should not require :a & :b as they are not in the result.#2021-04-0618:52nilernIndeed#2021-04-0713:44armedHey, have issue with malli.error/humanize, here is minimal example (cljs):
(let [Schema2 [:map
              [:field keyword?]]
      Schema [:map
              [:foo int?]
              [:bar [:or nil? Schema2]]]]
  (malli.error/humanize
   (malli/explain Schema {:foo 1
                          :bar {:field "test"}})))
Throws error: #object[Error Error: Vector's key for assoc must be a number.]
#2021-04-0713:46armedWhat I'm missing here?#2021-04-0713:48armedBy the way, in clojure I get another error for same snippet:
Execution error (ClassCastException) at malli.error/-ensure (error.cljc:124).
class clojure.lang.Keyword cannot be cast to class java.lang.Number (clojure.lang.Keyword is in unnamed module of loader 'app'; java.lang.Number is in module java.base of loader 'bootstrap')
#2021-04-0713:50ikitommi@armed it’s a known issue, see https://github.com/metosin/malli/pull/333#2021-04-0713:50ikitommiyou could say [:maybe Schema2] and it should work.#2021-04-0713:51ikitommibut, need to be fixed to be always robust.#2021-04-0713:52armed@ikitommi thanks for explanation, going to use :maybe#2021-04-0715:22danielnealWhen I call malli.util/update on a schema, the properties of the key I update are dropped, is there any way to preserve them?#2021-04-0715:24danielneal
(malli.util/update
    (malli.core/schema
      [:map
       [:a {:optional true}
        int?]
       [:b {:optional true} int?]])
    :a identity)
#2021-04-0715:24danielneal
;=> [:map [:a int?] [:b {:optional true} int?]]
#2021-04-0715:26nilernLooks like a bug to me :thinking_face:#2021-04-0715:27danielnealShall I open an issue?#2021-04-0715:35nilernthinking-face util/update is implemented by getting the value, then setting it But it would be reasonable to expect that update preserves the properties even if get + assoc would not#2021-04-0715:37danielnealI agree#2021-04-0715:38nilernI think an issue would at the very least be a better place to discuss how that should work#2021-04-0715:44danielnealcool, added one here https://github.com/metosin/malli/issues/412#2021-04-0720:11HankstenbergWhat would be a good way to "cut" a data structure using a schema that it partially adheres to? To keep what conforms to the schema and discard the rest? The best approach I can think of is to use m/explain and then use the data from :errors to run an update-in on the data. Is there a better way to do that? It seems to me like there should be.#2021-04-0720:21nilernSounds like a more general version of strip-extra-keys-transformer#2021-04-0720:23nilernOr perhaps the tentative coercer https://github.com/metosin/malli/issues/404#2021-04-0807:49ikitommi@roseneck if the strip-extra-keys-transformer is not enough (just map keys), please provide an example with schema, input and expected output.#2021-04-0812:56jeremieHi all, just wonder if there is a malli schema for validating reitit route? like the reitit.spec namespace in reitit.core but for malli. My use case is describing API endpoints and referencing the reitit route for each endpoint would be great. Thanks.#2021-04-0813:37ikitomminot yet. One of the initial reasons for creating malli was to have a tool where one can deep-merge nested schemas without tears, to be used in reitir route data validation.#2021-04-0813:37ikitommiwould you be interested in a PR?#2021-04-0816:56Setzer22Hi there! Missing spec's validation now that I've migrated to malli, I rolled my own thing: https://github.com/setzer22/malli-instrument Let me know what you think! 😄#2021-04-0817:29borkdudeI wonder, didn't malli already have something to instrument functions with malli schemas? if not so, how were the function specs intended to be used?#2021-04-0817:38Setzer22If there is, I couldn't find it anywhere 🤷 What's there, besides function schema syntax, is a global function registry and a macro, m/=> to add functions to that registry. I just built a super simple thing on top of all this#2021-04-0817:41ikitommi@setzer22 looks good. Plan is to have the instrumentation in malli core lib, but just not there yet, see: https://github.com/metosin/malli/issues/349#2021-04-0817:41ikitommiAlso related: https://github.com/metosin/malli/issues/369
#2021-04-1412:20Ian FernandezI've made some questions asking about Intrument impl#2021-04-1412:21Ian Fernandezwhat malli expects for it#2021-04-1413:11ikitomminoticed, thanks! need to think it over before answering.#2021-04-0817:45ikitommi... but the idea seems exactly what you have already implemented. Maybe the lib could be just a new ns under malli?#2021-04-0819:30Setzer22@ikitommi Oh, I'd be glad to contribute if you think it fits! 👍#2021-04-0819:35Setzer22let me know if you'd like me to submit a PR with this#2021-04-0910:57ikitommi@setzer22 I’ll draft my vision to the issue and then it’s all yours if you have time, planning to integrate the normal defn schema inferrer into it, needs var filtering etc. But, busy atm, I’ll link your lib to Malli README meanwhile, so people can pick it up.#2021-04-0910:59ikitommihttps://github.com/metosin/malli/commit/e57e0f546c096b8cb39b6b0d83c0ed3f9d555552#2021-04-0913:03Setzer22cool! I'll keep an eye on that, definitely interested 😄#2021-04-0913:03Setzer22also thanks for mentioning my lib in the readme 👍#2021-04-1017:41ikitommiSome breaking changes coming to the extender-api, moving -type and -type-properties from Schema to IntoSchema. And some new methods so that we can describe the Malli schemas with Malli:
(defprotocol IntoSchema
  (-type [this] "returns type of the schema")
  (-type-properties [this] "returns schema type properties")
  (-properties-schema [this] "maybe returns :map schema describing schema properties")
  (-children-schema [this] "maybe returns sequence schema describing schema children")
  (-into-schema [this properties children options] "creates a new schema instance"))
#2021-04-1017:48ikitommihttps://github.com/metosin/malli/pull/414#2021-04-1018:01ikitommialso, much better print presentations now:
(m/-enum-schema)
; => #IntoSchema{:type :enum}

(m/schema [*1 "A" "B"])
; => [:enum "A" "B"]

(type *1)
; => :malli.core/schema
#2021-04-1020:15HankstenbergIf I want to generate valid malli schemas of the form: [:map [:key1 int?] [:key2 int?] .... [:keyn int?]] using another malli schema - a meta-schema so to speak - what would that meta-schema look like? I've tried: [:tuple [:= :map] [:+ [:tuple simple-keyword? [:= :int]]]]) But it has a flaw: it e.g. generates: [:map [[:key1 int?] [:key2 int?] .... [:keyn int?]]] one pair of square brackets too much around the key-value-tuples. Does anybody know how to solve this using malli?#2021-04-1021:00ikitommi@roseneck it’s a sequence:
(mg/generate
  [:cat {:gen/fmap vec}
   [:= :map]
   [:* [:tuple :keyword :int]]]
  {:size 42, :seed 200})
;[:map
; [:IjPg2 -5908]
; [:E+:H9l* 287044292]
; [:*vCe._5z 10637946460]
; [:kLq67 -125]
; [:J9?D+FY4G 718]
; [:*_1*Rc+A -908]
; [:?qi 1523]
; [:x.+ -1]
; [:D4*5 -13]
; [:.C4 -80]
; [:r-A 3161894]
; [:rcC 1118]
; [:Q*._k*w8iG 194253522]
; [:o!dfA 21159377159]
; [:nx!h 973119]
; [:C:!Rw 109221]
; [:?N38.*3? -11854145]
; [:lb5Z?9V?_v* -42]
; [:i!M.l?D+ 2026049]
; [:++k43x -141680]
; [:U- -8]
; [:*x 1491554935632]
; [:heX2 1957851783]
; [:_PZ?q3+! -83]
; [:*H*x8N:.! 1452]
; [:k_Oy2+LXD -864224065]
; [:p 55]
; [:*o+1ul*5?? -1766412]
; [:W7:-61 -17174129950]
; [:qv79TTj_+ 21380662]
; [:h -30089]]
#2021-04-1021:04HankstenbergThanks a lot! 🙂#2021-04-1210:16Ian FernandezHi, I've asked some questions at a malli issue to understand some design motivations 🙂 https://github.com/metosin/malli/issues/349#2021-04-1210:17Ian FernandezShould be good if anyone here could contribute 🙂#2021-04-1219:50jcfHi all! 👋 I'm looking into specifying higher-order functions, and wonder if there's a schema that resembles Clojure's ifn?. I notice there's no ifn? predicate in the predicate-schemas (nor does fn? make an appearance), so I'm leaning towards defining something in user space (assuming there's nothing in Malli that I've missed?). I'm hesitant to do something like [:alt keyword? [:=> :cat any?]] because I'm creating a closed schema that would need to be extended for any IFn that might crop up in future… thinking of a custom IntoSchema or similar maybe. Any tips are much appreciated!!#2021-04-1220:08jcfI can do this:
[:fn ifn?]
#2021-04-1220:08jcfI'm just not sure if that's a bad way to go. 🙂#2021-04-1304:15ikitommithat's the way to do it now.#2021-04-1310:23jcfThanks, @U055NJ5CC!#2021-04-1308:17Ben SlessIs it by design that schema transformations like mu/assoc don't play well with an unregistered schema?#2021-04-1309:06ikitomminot by design, could you repro if there is something that doesn’t work?#2021-04-1309:28Ben Sless
(def Foo
  [:map
   [:a int?]])

(mu/assoc Foo :b ::bar)
Minimal example
#2021-04-1309:47ikitommiyou should declare the ::bar so that is is visible, either: 1. override the default registry 2. pass the registry into Foo when creting it (it closes over the creation time registry) 3. pass the registry into mu/assoc
(def registry
  (merge (m/default-schemas) {::bar int?}))

(def Foo
  (m/schema
    [:map
     [:a int?]]
    {:registry registry}))

(mu/assoc Foo :b ::bar)
;[:map 
; [:a int?] 
; [:b :user/bar]]

(mu/assoc [:map [:a int?]] :b ::bar {:registry registry})
;[:map 
; [:a int?] 
; [:b :user/bar]]
#2021-04-1309:54Ben SlessI managed to get myself into this corner like so: • wanted content dependent schema • wanted to parametrize the schema (makes it extensible) • figured out I'd do it by delaying registry building and schema compilation to run-time. • With registry I need ::my-schema • Can't transform anything with ::my-schema at compile time I can create a placeholder registry for it but it seems like it would lead to errors down the line#2021-04-1309:57Ben SlessI'd be happy to adopt a better idea#2021-04-1310:00ikitommime too 🙂 spec partially checks the references eagerly, partially lazily (e.g. s/keys), malli is currently eager.#2021-04-1310:01ikitommithere is an internal escape hatch: :ref doesn’t check the reference if :malli.core/allow-invalid-refs option is truthy.#2021-04-1310:01ikitommiit is used with local registries, which can have… holes.#2021-04-1310:03ikitommi
(m/validate
  [:schema {:registry {::foo [:ref ::bar]}} ;; incomplete registry
   [:tuple {:registry {::bar int?}}
    ::bar ::foo]]
  [1 2])
; => true
#2021-04-1310:06ikitommiideas welcome how to make this good.#2021-04-1310:12Ben SlessI don't know if it's good, but perhaps a :delay or :defer schema, which delays registry lookup to validation, with ample warning, care, etc.#2021-04-1310:18Ben SlessPerhaps even wrap it in a function which will always emit warnings when it's called, i.e.
(mu/assoc Foo :b (m/schema (m/defer ::bar)))
STDERR: Deferred Warning *at* - instances of deferred schema must be provided with a registry at run time!
You can also throw when instantiating an explainer, transformer, or validator from it, which is when you actually need the registry
#2021-04-1310:21ikitommicould it be just [:ref {:lazy true} ::bar]?#2021-04-1310:21Ben SlessAh, laziness has to be explicit#2021-04-1310:21ikitommioh, ref’s are lazy already :thinking_face:#2021-04-1310:22Ben Slessyeah, this didn't work 🙂#2021-04-1310:22Ben Slesswe need lazier laziness#2021-04-1310:23ikitommitry (m/-lazy ::bar options)#2021-04-1310:24ikitommirefs resolve eager by default, but one can create lazy refs with that.#2021-04-1310:24ikitommi
(let [-ref (or (and lazy (-memoize (fn [] (schema (mr/-schema (-registry options) ref) options))))
                      (if-let [s (mr/-schema (-registry options) ref)] (-memoize (fn [] (schema s options))))
                      (when-not allow-invalid-refs
                        (miu/-fail! ::invalid-ref {:type :ref, :ref ref})))
#2021-04-1310:24Ben SlessCool, it worked 🙂#2021-04-1310:24Ben SlessAlways good to know some black magic#2021-04-1310:25ikitommicould make a version of that which doesn’t require the options.#2021-04-1310:25Ben SlessIt makes me wonder why [:ref {:lazy true} ::bar] didn't work#2021-04-1310:26ikitommiit’s a property of the IntoSchema, not Schema instance.#2021-04-1310:27ikitommiby design, all the IntoSchemas are crated using a function, which can take properties how the IntoSchema works. Easy to extend the system that way and DCE drops all the unneeded schemas.#2021-04-1310:27ikitommifor example, it’s reletively easy to create custom collection schema types:
(defn -collection-schema [{type :type fpred :pred, fempty :empty, fin :in :or {fin (fn [i _] i)} :as opts}] ...)
#2021-04-1310:28ikitommi:ref has:
(defn -ref-schema
  ([]
   (-ref-schema nil))
  ([{:keys [lazy type-properties] :as opts}] ...))
#2021-04-1310:28ikitommibut, could lift the lazy into a :ref schema property too. so one can say [:ref {:lazy true} ::bar] as data.#2021-04-1310:29ikitommiif you need that, please write an issue.#2021-04-1310:32Ben SlessThank you!#2021-04-1310:33Ben SlessI wonder if I should settle for m/-lazy#2021-04-1310:33Ben SlessIf I should consider functions prefixed with - as implementation detail, then I'd say that I shouldn't and open that issue#2021-04-1310:34ikitommithings starging with - are ok to use: https://github.com/metosin/malli#alpha#2021-04-1310:38Ben Sless> might evolve during the alpha That's a risk I'm willing to take. I think if m/-lazy develops in any direction it won't be one which will have friction with what I'm trying to do, on the contrary. Thanks again for the help and guidance, you rock#2021-04-1309:30Ben SlessAnother issue I managed to stumble on, I defined a dependent schema like https://github.com/metosin/malli#content-dependent-simple-schema It works well but throws when I pass it to reitit routes when the coercion is compiled#2021-04-1309:34Ben SlessGreat, now I'm unable to reproduce it 😞#2021-04-1309:47ikitommiyou should declare the ::bar so that is is visible, either: 1. override the default registry 2. pass the registry into Foo when creting it (it closes over the creation time registry) 3. pass the registry into mu/assoc
(def registry
  (merge (m/default-schemas) {::bar int?}))

(def Foo
  (m/schema
    [:map
     [:a int?]]
    {:registry registry}))

(mu/assoc Foo :b ::bar)
;[:map 
; [:a int?] 
; [:b :user/bar]]

(mu/assoc [:map [:a int?]] :b ::bar {:registry registry})
;[:map 
; [:a int?] 
; [:b :user/bar]]
#2021-04-1310:14yuhanAre there built-in functions to throw errors on invalid input? The plans for instrumentation in the above issue are nice, but I'm looking for something simple like spec/assert#2021-04-1310:38nilernSeems not#2021-04-1310:40nilernYou can always (assert (thingy-validator dada)) but the error is not so useful#2021-04-1310:43yuhanYeah, I wrote my own for now:
(defn malli-assert
  ([schema value]
   (malli-assert schema value ""))
  ([schema value msg]
   (when-not (malli/validate schema value)
     (throw (ex-info (clojure.string/join "/n"
                       (cons msg
                         (flatten
                           (malli.error/humanize
                             (malli/explain schema value)))))
              {:value value})))))
#2021-04-1310:44nilernMake a PR?#2021-04-1310:44yuhanIt's not ideal because humanize returns nested messages according to the path of the error, which I just flatten into a single string#2021-04-1310:45yuhanOk I'll submit an issue, just wanted to check if it was a design decision not to have an assert#2021-04-1310:47nilernMaybe AssertionError would be more appropriate :thinking_face:#2021-04-1310:50nilernAnd maybe use *assert* and make it a macro#2021-04-1310:50nilernSpec assert seems to use ex-info and a separate *assert* equivalent var#2021-04-1310:28jcfI have a follow up question from https://clojurians.slack.com/archives/CLDK6MFMK/p1618257034389400 regarding emitting configuration for clj-kondo to pick up (which is an awesome feature by the way!). I have schematised the following code:
(m/=> hash-map-by
  [:=> [:catn [:f [:fn ifn?]] [:coll coll?]] map?])

(defn hash-map-by
  "Returns a map of the items in `coll` keyed by `f`."
  [f coll]
  (into {} (map (juxt f identity)) coll))
The function takes an arbitrary function, f, and a collection that will be converted into a map by applying f and identity to each item in the collection. Pretty standard stuff. 🙂 When I emit clj-kondo config with (mc/emit!), I get the following EDN:
{:lint-as #:malli.schema{defn schema.core/defn},
 :linters {:type-mismatch {:namespaces {example.hash-map {hash-map-by {:arities {2 {:args [:fn :coll], :ret :map}}}}}}}}
Please note, the 2-arity args say :fn and :coll returning a :map which means I get linting issues with something like (hash-map-by :user/id [{:user/id 1} {:user/id 2}]). Is this a bug worthy of a pull request or am I once again demonstrating my naivety? 🙈
#2021-04-1310:32ikitommicurrently there is no way to override per schema instance how the clj-kondo works, but would be easy to add. also, having an ifn? schema built-in, it could have the correct clj-kondo type. interested in a PR?#2021-04-1310:32ikitommifor the latter that is.#2021-04-1310:32ikitommifor the first, for the second, something like:#2021-04-1310:33ikitommi
[:fn {:clj-kondo/type :ifn} ifn?]
#2021-04-1310:34jcfI'm very interested in implementing this as we'd need it to complete the replacement of clojure.spec with Malli in our codebase, I think.#2021-04-1310:34jcfI can create a PR for sure. 💯#2021-04-1311:00jcf@U055NJ5CC can I just clarify what you're thinking in terms of a PR, please? I can add #'ifn? to the predicate-schemas and then these tests pass:
(testing "ifn schemas"
    (let [schema (m/schema ifn?)]
      (is (true? (m/validate schema (fn []))))
      (is (true? (m/validate schema (constantly 1))))
      (is (true? (m/validate schema :keyword)))
      (is (true? (m/validate schema (reify clojure.lang.IFn
                                      (invoke [_] "Invoked!")))))))
Is that what you had in mind when you mentioned having an ifn? schema built in?
#2021-04-1311:09ikitommiyes, but also mappings for transformers, generators, json-schema, humanized errors and clj-kondo.#2021-04-1311:11jcfI'll take a look at implementing the full feature set for the ifn? domain. 👍#2021-04-1311:12jcfI'll not implement proper generation of interesting functions. Don't want to put us all out of a job. 😉#2021-04-1311:21jcfOh, I think I see what you mean. You want ifn? to be parameterised so you can schematize the args and return values…?#2021-04-1311:22jcfSo in the schema generator you can do something more than this:
(defmethod -schema-generator 'ifn? [_ _] (gen/return ::ifn))
#2021-04-1311:23jcfI'd need to generate a function that returns valid data given valid arguments.#2021-04-1311:23ikitommireally, why?#2021-04-1311:24ikitommithere is already :=> and :function which have proper input & output generators#2021-04-1311:24jcfBecause I thought that was what you wanted. 🙂#2021-04-1311:24jcfI think I misunderstood.#2021-04-1311:24ikitommino, just the simple thing, lke fn? but bit different 🙂#2021-04-1311:24jcfIf ifn? can remain a simple predicate, I should be able to have a first pass at a PR before the lunchtime walk. 🙂#2021-04-1311:24ikitommi👍#2021-04-1311:47jcfOne pull request with my stab at adding ifn? to Malli's list of supported predicates. Gotta go for a lunchtime stroll with the pup; I'll check in when I'm back in about an hour. 🙇 https://github.com/metosin/malli/pull/416#2021-04-1310:38nilernSeems not#2021-04-1311:47jcfOne pull request with my stab at adding ifn? to Malli's list of supported predicates. Gotta go for a lunchtime stroll with the pup; I'll check in when I'm back in about an hour. 🙇 https://github.com/metosin/malli/pull/416#2021-04-1318:58yuhanIs it possible for schemas to self-reference? Naively trying to make a recursive schema causes a stack overflow:
(malli/schema
  ::tree
  {:registry (merge (malli/default-schemas)
               {::node :int
                ::tree [:or
                        ::node
                        [:tuple ::tree ::tree]]})})
 
#2021-04-1319:00ikitommi@qythium use the :ref, luke:
(mg/generate
  (m/schema
    ::tree
    {:registry (merge (m/default-schemas)
                      {::node :int
                       ::tree [:or
                               ::node
                               [:tuple [:ref ::tree] [:ref ::tree]]]})})
  {:seed 3})
; => [[-26764 [[1 73136] [13307055 -1]]] [-381 [[-3 587742] -243724556]]]
#2021-04-1319:01yuhanawesome, thanks!#2021-04-1319:28yuhanSo schema references only in :ref and :map have to be qualified keywords? Strings appear to work too, but not plain keywords#2021-04-1319:36yuhanSeems like it, I found malli.core/-reference? in the source code which checks for qualified-keyword? or string?, but this doesn't seem to be documented#2021-04-1319:31yuhanAlso it seems that the ::foo shorthand syntax doesn't work on the http://malli.io playground#2021-04-1319:37ikitommiyes, references should be qualified keywords or strings. http://malli.io … must be a sci-thing.#2021-04-1319:45yuhanOk, filed an issue on the http://malli.io github#2021-04-1319:49yuhanTrying to use unqualified keys as references tripped me up quite a bit at the beginning, since this requirement isn't documented and the error message just says :malli.core/invalid-schema#2021-04-1319:50ikitommidoc enhancement PRs are most welcome.#2021-04-1319:51ikitommi(the error keyword could be better here)#2021-04-1319:57yuhanI'll just file issues for now if that's ok - still in the early stages of experimenting with the library and not confident of writing docs#2021-04-1319:59yuhanAnother strange thing I encountered:
;; This works as a schema
[:map {:registry {::foo :int}}
 ::foo]

;; so does wrapping the keyword in a vector
[:map {:registry {::foo :int}}
 [::foo]]
;; to pass it options
[:map {:registry {::foo :int}}
 [::foo {:optional true}]]


;; These are ok too
[:map {:registry {::foo [:tuple :int :int]}}
 ::foo]
[:map {:registry {::foo [:tuple :int :int]}}
 [::foo {:optional true}]]


;; But not this??
(malli/schema
  [:map {:registry {::foo [:tuple :int :int]}}
   [::foo]])
;; => Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:16).
;;    :malli.core/invalid-schema {:schema [:tuple :int :int]}
#2021-04-1404:27ikitommi@qythium looks like a bug in parsing entries.#2021-04-1404:27ikitommiweird thing that haven’t bumped into it before.#2021-04-1408:27Setzer22I was wondering, is there dedicated support in function schemas to handle things like (fn [a, b, & {:keys [c, d]}) I'd like to be able to do: (m/=> fn-name [:=> [:cat Foo, Bar, [:map?? [:c Baz] [:d Other]]]]#2021-04-1408:29Setzer22I couldn't find something to put in place of :map?? in my example#2021-04-1409:08nilernI don't think so You could do [:cat Foo, Bar, [:* [:alt [:cat [:= :c] :any] [:cat [:= :d] :any]]]] (permits duplicates, but so does the fn itself)#2021-04-1409:10nilernIt would be good to have something more convenient especially with Clojure 1.11: https://clojure.org/news/2021/03/18/apis-serving-people-and-programs#2021-04-1409:13nilernLike [:cat Foo, Bar, [:alt [:* [:alt [:cat [:= :c] :any] [:cat [:= :d] :any]]] [:map [:c {:optional true} :any] [:d {:optional true} :any]]] or something can handle the 1.11 feature as well but who likes to write it even once#2021-04-1409:18nilernYou could write your own utility fn to generate that but really it should be built in to the fn schemas somehow#2021-04-1409:19ikitommimaybe: [:cat Foo, Bar, [:& [:c Baz] [:d Other]]]?#2021-04-1409:27nilernBut it does not do normal varargs so :& could be confusing. :&n?#2021-04-1617:35schmeeis there a way to use Malli to remove nil values in a map? :thinking_face:#2021-04-1620:34Leonid KorogodskiThe oddest thing. I have two projects. Both use malli 0.2.1 (according to `lein deps :tree`). But one of them accepts the syntax `[:map [:a string?] [:b string?]]` just fine, while the other throws an error on `(restart)`:
data-spec collection [] should be homogeneous, 3 values found
Any idea what could be the cause?
#2021-04-1620:35borkdude@lkorogodski Can you try with lein clean and then run your project again?#2021-04-1620:35borkdudeYou might have some left-overs in your target folder or something#2021-04-1620:36Leonid KorogodskiSame result.#2021-04-1620:57borkdude@lkorogodski Please don't paste such long output into slack but rather use something like github gist#2021-04-1620:57borkdudeWhat happens when you do:
(require '[ :as io])
(io/resource "malli/core__init.class")
#2021-04-1620:58borkdudejust checking if this isn't an AOT-related problem#2021-04-1622:21Leonid KorogodskiOk, sorry. Just a moment.#2021-04-1622:27Leonid Korogodski
nREPL server started on port 64815 on host 127.0.0.1 - 
Connecting to local nREPL server...
Clojure 1.10.3
(require '[ :as io])
=> nil
(io/resource "malli/core__init.class")
=> nil
(require '[malli.core :as m])
=> nil
(m/validator [:map [:a string?] [:b string?]])
=>
#object[malli.core$_map_schema$reify$reify__21001$fn__21022
        0x5361e937
        "
That seems to work. But when I call (restart), it fails. The usage for the router is as follows:
["/my-route" {:get {:summary    "..."
                    :parameters {:header {:authorization string?}
                                 :query  [:map
                                           [:a string?]
                                           [:b string?]}
                    ;; other things
                   }}]
#2021-04-1622:33Leonid Korogodskimap? works in place of [:map ...] but doesn't check enough, of course.#2021-04-1702:23Leonid Korogodski
(defn restart []
  (mount.core/stop)
  (mount.core/start))
#2021-04-1712:14ikitommi@lkorogodski in your example, there are both clojure specs (`:authorization)` and malli schemas (`:query)`. You should pick either one via :coercion option in the route data and use it in all the places. The error comes from spec-tools, so I believe you have coercion set to clojure.spec.#2021-04-1712:14ikitommithere are malli examples under examples . Hope this helps you to the right direction#2021-04-1712:17ikitommi@schmee sure, nothing built-in I guess, but you can add a custom decoder into the :map schema, reduce over the all map entries and remove nils.#2021-04-1712:18ikitommithere are similar imps in malli.transform for stripping out extra keys and transforming keys.#2021-04-1712:51Leonid KorogodskiThanks!#2021-04-1713:12ikitommilooking at GraphQL fragments, would be relatively easy to support those in malli. But not sure if that would be a good idea. Maybe not.#2021-04-1719:04eskosIIRC they're very close to Turing complete, so maybe not 🙂#2021-04-1719:06eskosThere's https://github.com/Vincit/venia which is data based so maybe have venia/Malli thingy instead? Would save you from implementing your own GraphQL parsing stuff as well.#2021-04-1713:13ikitommiwriting GraphQL<->Malli would be easier with that for sure.#2021-04-1906:57Hugh PowellI'm trying to create a schema for a CSV data structure with defined, case insensitive headers. It must also be able to generate a legal data structure. I'm using the same pattern as clojure.data.csv, a vector of vectors with the first vector representing the headers and the remaining vectors representing the rows. I can create a schema for the headers that checks validity fairly simply
(def headers
  [:cat
    [:re "(?i)header1"]
    [:re "(?i)header2"]
    ...
    [:re "(?i)headerN"]])
But test.chunk doesn't handle flags (the (?i) bit), so I don't have a generator. My next guess is to create a schema that can match a given string case-insensitively and then a generator that will generate a randomly cased version of that string. I can write these two functions, but can't work out how to create a schema with them. Any thoughts or docs you can point me to?
#2021-04-1907:49ikitommi@hugh336 maybe a custom generator? [:re {:gen/gen gen/alphanumric} "(?i)header1"]#2021-04-1909:02Hugh PowellAha! Awesome, thanks for the quick reply :thumbsup:#2021-04-1909:22ikitommihere a {:gen/elements ["header1"]} would be good, just data and always correct.#2021-04-1909:22ikitommihttps://github.com/metosin/malli#value-generation#2021-04-1914:31bartuka@ikitommi I also have several malli schemas that need to verify for [:fn (complement str/blank)] and I saw an open discussion from some years ago in Malli about adding validation of stripped strings to [:string] base schema#2021-04-1914:31bartukais this still desired?#2021-04-1914:32bartukanot some years.. last year, sorry. thought was 2019 https://github.com/metosin/malli/pull/205#2021-04-1914:40ikitommiis the [:string {:min 1}] bad?#2021-04-1914:40ikitommimaybe (def NonEmptyString [:string {:min 1}]) liike we would do in Schema?#2021-04-1914:42ikitommior:
[:map {:registry {::non-blank-string [:string {:min 1}]}}
 [:name ::non-blank-string]
 [:address [:map [:street ::non-blank-string]]]]
#2021-04-1915:02bartuka(m/validate [:string {:min 1}] " ") ;; => true but I expect false if no blank is allowed#2021-04-1915:03bartukaI like the idea of using {:trim true :min 1 :max 99} .. looks very clean and we dont need to add custom registries#2021-04-2008:03SathiyaIs there an option to support blank string in :enum schema. :maybe supports nil but not blank values. currently i used [:or empty? [:enum "foo" "bar"]] to do this validation but is there a way to create a optional-enum registry that supports this scenario. The problem I'm facing using :or is I'm getting two error messages for each options. It would be better to have one message that says, value can be blank or either one of the valid values#2021-04-2008:07delaguardo[:enum "" "foo" "bar"] should work https://malli.io/?value=1&amp;schema=%5B%3Aenum%20%22%22%20%22foo%22%20%22bar%22%5D#2021-04-2008:08delaguardohowever error message will hide empty string#2021-04-2008:08delaguardo["should be either , foo or bar"]#2021-04-2010:24SathiyaThanks @U04V4KLKC#2021-04-2011:55anonimitorafHi guys, I came across https://github.com/metosin/malli/issues/125 when looking for ways to "instrument" functions. My question is, how do you guys "instrument" functions with malli at the moment?#2021-04-2018:12Ivan FedorovHey folks 👋 Are you aware of any generators built on malli ? Interested in crux pull vector, pathom resolvers, entity constructors, fulcro queries#2021-04-2021:03ikitommicommented on the instrument issue, sorry for the lag.#2021-04-2021:03ikitommihttps://github.com/metosin/malli/issues/349#issuecomment-823596409#2021-04-2106:38anonimitorafThanks for the update! Looks good!#2021-04-2021:04ikitommi@d.ian.b do you mean value generators or schema generators (providers)? Both would be cool. plan is to support EQL for slicing schemas at some point.#2021-04-2022:08refsetUhh, I think the wrong I.F. 🙂 /cc @U0A5V8ZR6#2021-04-2106:51Ivan FedorovI mean codegen. I want to describe malli schema and then have all the goodies (pull vectors, pathom resolvers, etc) generated. I may end up doing it myself, just asking if you have seen anything like this already.#2021-04-2108:03ikitommidon’t think such things exist yet, go for it 🙂 the current malli lib has a lot of optional ns’s for external things (json schema, dot, etc), not sure where the limit should be - what optional things should be in the core lib (as optional namespaces) and which should be separate libs. At least EQL<->Malli could be in the core. Or then just redesign the code into a monorepo like reitit and make all the extras as separate modules.#2021-04-2202:59rutledgepaulvI don't mind things being in core if they don't require dependencies but if it's going to bloat the malli dep tree would prefer separate modules so I can keep my projects lean. Maybe that's an easy way to draw the line?#2021-04-2614:28Ian Fernandez@U28A9C90Q @U3Y18N0UC#2021-04-2615:47marciolTo me seems that at first glance it'd be better to separate all functionality that isn't essential to make Malli works in separated libs, but if we think about Malli as a full experience in terms of how to deal with schemas - and I hope that it became the best in class tool to work with external data, to such a point that people of other ecosystems really envy the Malli experience - I think that the full-batteries included is a better approach, so I really don't mind if all those niceties are included on Malli itself.#2021-04-2021:07ikitommioh, and merged 414, breaking change for the extender api. But schemas can now be described with schemas (just need to describe all schemas first :)), the new IntoSchema protocol:
(defprotocol IntoSchema
  (-type [this] "returns type of the schema")
  (-type-properties [this] "returns schema type properties")
  (-properties-schema [this] "maybe returns :map schema describing schema properties")
  (-children-schema [this] "maybe returns sequence schema describing schema children")
  (-into-schema [this properties children options] "creates a new schema instance"))
#2021-04-2021:08ikitommialso, most malli.utilfunctions got bit faster as schema instances now have a direct handle to parent, no need to create new instances. which is nice.#2021-04-2021:09ikitommie.g. ´m/type` is looked via the parent:
(defn type
  "Returns the Schema type."
  ([?schema]
   (type ?schema nil))
  ([?schema options]
   (-type (-parent (schema ?schema options)))))
#2021-04-2021:09ikitommiso, the public api remains the same.#2021-04-2108:03ikitommidon’t think such things exist yet, go for it 🙂 the current malli lib has a lot of optional ns’s for external things (json schema, dot, etc), not sure where the limit should be - what optional things should be in the core lib (as optional namespaces) and which should be separate libs. At least EQL<->Malli could be in the core. Or then just redesign the code into a monorepo like reitit and make all the extras as separate modules.#2021-04-2109:32nilernLet's not forget https://github.com/kwrooijen/gungnir#2021-04-2112:00andrea.crottiI'm just trying malli for the first time today, and it looks pretty great already#2021-04-2112:01andrea.crottione thing I didn't quite understand though, if I annotate a function with (m/=> ... is there a way to actually force that check at runtime to make it fail when the schema doesn't match? I only see that related with generating a clj-kondo config, which is great but I thought that run-time assertions would be also there, even if disabled by default maybe#2021-04-2112:06nilernhttps://github.com/metosin/malli/issues/349 but @setzer22 has a working implementation https://github.com/setzer22/malli-instrument#2021-04-2112:09Setzer22Yes, I've been using my implementation for a couple of weeks now 👍 The base features are working, but sometimes malli has trouble generating human readable descriptions of errors#2021-04-2112:09Setzer22if you give it a try, please report any issues!#2021-04-2112:14andrea.crottiah nice, yeah no rush was just wondering if I missed something from the docs#2021-04-2112:42ikitommi@setzer22 yes, there is an open PR about robust humanized errors. Few combinations that don’t work, but the idea to fix it seems legit.#2021-04-2112:54Setzer22@ikitommi Good to know!#2021-04-2113:15andrea.crottianother thing I noticed is that there isn't a way to just throw an exception if something fails to validate. I ended up with something like
(let [my-map ...
       errors (m/explain schema my-map)]
  (if (nil? errors) my-map (throw (ex-info "validation failed" errors)))
but isnt't there an easier way to do that?
#2021-04-2113:22nilernWe should have an assert...#2021-04-2113:27nilernThere wasn't even an issue so I made one https://github.com/metosin/malli/issues/420#2021-04-2113:56nilernHmm hard to make an efficient assert that makes the validator and explainer behind the scenes. Especially Cljs would be tricky.#2021-04-2115:47andrea.crottimm maybe you don't always need the explainer#2021-04-2115:47andrea.crottiwould be nice I guess but not sure it needs to be the default#2021-04-2115:56nilernDropping the explainer does not make it any easier.#2021-04-2117:34yuhanThanks for creating the issue! it must have slipped my mind. Do you mean caching a validator at the assert's compile time? I would be worried about the validator then going out of sync when you redefine the schema during dev time#2021-04-2117:38yuhanMaybe the *assert* dynamic var could have 3 levels: • runtime validate (slowest but correct) • compile-time validator (more efficient, use if schemas are fixed) • compiles to nil#2021-04-2213:08nilernYeah I figured you just forgot. And I did mean the assert macro would create the validator#2021-04-2213:10nilernIt is probably ok to not do that in dev mode although sometimes assertion overhead is too much even then#2021-04-2201:12ZaymonHey all. I might be missing something essential here but I don’t fundamentally understand why this won’t compile:
(defn longer-than-5? [x] (<= 5 (count x)))
(def MySchema [:map [:some-key [:and string? longer-than-5?]]])

; :malli.core/invalid-schema {:schema #function[example/longer-than-5?]}
Is there something special about string?, int? ect that makes them more than just a function from a -> bool ?
#2021-04-2201:36ZaymonLooks like I need to specify the function with [:fn longer-than-5?] !#2021-04-2213:12nilernWhat is special is that they are in the m/predicate-schemas registry which is merged into the default registry#2021-04-2301:37Zaymon@U4MB6UKDL That’s interesting. Thank you#2021-04-2209:21Yevgeni TsodikovHi, I think the strip-extra-keys-transformer removes mandatory fields if there are in an [:or ... ] vector:
(def my-schema [:and
                [:map [:a int?]]
                [:or
                 [:map [:b int?]]
                 [:map [:c int?]]]])
=> #'...
(m/validate my-schema {:a 1})
=> false
(m/validate my-schema {:a 1 :b 1 :c 1})
=> true
(m/decode my-schema {:a 1 :b 1 :c 1} (mt/transformer mt/strip-extra-keys-transformer))
=> {:a 1}
Did I configure the schema wrong? How can I work around it?
#2021-04-2209:25ikitommimaybe:
[:and
 [:map 
  [:a int?]
  [:b {:optional true} int?]
  [:c {:optional true} int?]]
 [:fn {:error/messag "only :a or :b is allowed"}
  (fn [{:keys [b c]}] (not (and b c)))]]
or:
[:or
 [:map
  [:a int?]
  [:b int?]]
 [:map
  [:a int?]
  [:c int?]]]
would those work for you?
#2021-04-2209:25Yevgeni TsodikovChecking, I want to see how they integrate with reitit ’s swagger docs#2021-04-2209:26ikitommiyou can run from repl:
(malli.swagger/transform
  [:and
   [:map 
    [:a int?]
    [:b {:optional true} int?]
    [:c {:optional true} int?]]
   [:fn {:error/messag "only :a or :b is allowed"}
    (fn [{:keys [b c]}] (not (and b c)))]])
to see what comes out.
#2021-04-2209:27ikitommiswagger doesn’t support anyOf JSON Schema, so the latter will not show in swaggr docs.#2021-04-2209:27ikitommiopenapi has that, but not integrated into reiti.#2021-04-2210:26Yevgeni TsodikovI’ve used the first option, it worked for me. Thanks!#2021-04-2209:51DosHi @ikitommi I am facing interesting behavior with explain. I don't understand why order matters...#2021-04-2209:53Dos
(def params-function?
  (malli/schema
   [:function
    [:=> [:cat map? [:maybe map?] map?] map?]]
   {::malli/function-checker malli.generator/function-checker}))

(def next-function?
  (malli/schema
   [:function
    [:=> [:cat map? [:maybe map?] map?] keyword?]]
   {::malli/function-checker malli.generator/function-checker}))
#2021-04-2213:47ikitommi@U0D8J9T5K oh, that’s bad. will fix it.#2021-04-2214:52ikitommiFixed in master: https://github.com/metosin/malli/pull/421#2021-04-2417:50ikitommi:gen/schema … this might be useful:
(mg/sample [:any {:gen/schema :int}])
; => (0 -1 -2 1 0 0 18 1 -3 -1)

(mg/sample [:string {:gen/schema [:int {:gen/min 10, :gen/max 100}], :gen/fmap 'str}])
; => ("11" "11" "11" "10" "10" "10" "10" "13" "13" "66")
#2021-04-2707:20Adam HelinsAfter WASM, I am now describing a Lisp using Malli and once again am doing heavy use of recursive generation. I'm probably gonna be spamming a few questions in the following days. That kind of :set are always empty, am I doing something wrong with :ref? No such problem with :vector.
(malli.gen/sample [:set
                   {:registry {::foo :int}}
                   [:ref ::foo]])
#2021-04-2711:44Adam HelinsIs there any design decision behind not generating NaN and Infinity?
(defmethod -schema-generator :double [schema options] (gen/double* (merge (-min-max schema options) {:infinite? false, :NaN? false})))
#2021-04-2711:45Ivan FedorovAny good intro? I’m playing with schemas, and can’t see a formal definition of it in README Currently I’m thinking that [:schema] schemas must have a :registry followed by a single root subschema reference. I’ve watched the talk by Tommi Reiman from London Clojurians. It’s helpful, but I’m lacking some formal learning material. Thanks in advance!#2021-04-2712:10Ivan FedorovI’m trying to understand the best way to describe the entity types of an app. Probably it’s just a single big registry then and multiple (def schema:entity-type-n) using it.#2021-04-2718:48ikitommi@ognivo most malli-based projects I’ve seen use a mutable global registry with custom register function like spec (README has a guide how to set that up), registering looks like:
(register :user/id :int)
you can also just define schemas as var in schema style, using the default immutable regisitry:
(def Userid :int)
there was an Entities and Values guide in the README, but it was removed.
#2021-04-2815:23Ben SlessIt's also possible to delay schema compilation and registry instantiation to start-up time. I built a registry dynamically and injected it into a Component system, the registry itself contained content dependent schemas which depended on input arguments. It was fun. I might cajole my employer to let write something about it#2021-04-2816:52ikitommilooking forward to that!#2021-04-2811:39pithylessI’m reworking some error handling - especially dev vs prod. What’s the story with virhe? Is there a version of that coming soon? Or perhaps even some reflections and retrospectives that we could apply to our own work?#2021-04-2816:57ikitommi@pithyless there is a unpublished copy of reitit-error-pretty-printer in malli. should extract the common parts of malli + reitit and push out to virhe. to be honest, might be better if virhe was a community owned clj-commons library, not ours. there would be a lot to do with colors, themes, cljs-support etc, which needs work, which needs time, which we don’t have that much extra. happy to contribute code, ideas and requirements to that. For now: just copy the existing reitit thing if you want the looks.#2021-04-2817:16pithylessare you referring to this? https://github.com/metosin/reitit/blob/master/modules/reitit-dev/src/reitit/dev/pretty.cljc It'd be great to read about the ideas and lessons learned on the virhe README if you find some free time to catch your breath. Either way, thanks for all the ceaseless work and FOSS publishing! 🙂#2021-04-2816:58ikitommiI’ll write the ideas and lessons learned to virhe README, soon. Might have time to push out first version before summer vacations.#2021-04-2817:01ikitommi🥳 pushed just out [metosin/malli "0.5.0"] - small fixes & improvements and the ability to describe Malli Schema Syntaxes with Malli. Small breaking changes for library extenders, more info in the CHANGELOG: https://github.com/metosin/malli/blob/master/CHANGELOG.md#050-2021-04-28#2021-04-2821:29eoliphantHi, dunno if I’m doing something wrong, but I’m having issues with local/custom registries and qualified keyword entries. Validation works fine, but getting properties, the original form back, etc don’t seem to work
(def registry*
  (merge
    (m/default-schemas)
    {:testfoo [:string {:a :b}]
     :test/bar [:string {:c :d}]
     ::testbaz [:string {:a :b}]}))
(m/properties :testfoo {:registry registry*})
; => {:a :b}
(m/properties ::testbaz {:registry registry*})
; => nil
#2021-04-2904:47ikitommi@eoliphant What happens is: • :testfoo is not a qualified key, it’s contents are inlined • ::testbaz is a qualified = schema reference, instead of the inlined contents, you get the reference back. References are internally of type :malli.core/schema , which can also have properties. To get the actual schema behind the reference, you can m/deref the schema. • analogy is to Vars in Clojure, e.g. you get #'inc (var) back, not inc (function value)
(def registry*
  (merge
    (m/default-schemas)
    {:testfoo [:string {:a :b}]
     :test/bar [:string {:c :d}]
     ::testbaz [:string {:a :b}]}))

(m/type :testfoo {:registry registry*})
; => :string

(m/type ::testbaz {:registry registry*})
; => :malli.core/schema

(-> ::testbaz
    (m/schema {:registry registry*})
    (m/deref))
; => [:string {:a :b}]

(-> ::testbaz
    (m/schema {:registry registry*})
    (m/deref)
    (m/properties))
; => {:a :b}
#2021-04-2904:49ikitommithat said, not happy how the references work atm, in most malli-based codebases I’ve seen, there is a lot of manual m/deref / m/deref-all calls to inline things. Also, the current behavior is not documented properly. Not sure what would be a correct way to handle these. Ideas welcome.#2021-04-2904:56ikitommicompared to spec, any lookup to a registry pulls the actual value, but when the reference is part of some other specs form, it’s kept as it is;
(require '[clojure.alpha.spec :as s])

(s/def ::kikka int?)

(s/form ::kikka)
; => clojure.core/int?

(s/form (s/tuple ::kikka))
; => (clojure.alpha.spec/tuple :user/kikka)
#2021-04-2905:00ikitommithis sums the current functionality:
(m/schema [:tuple :testfoo ::testbaz] {:registry registry*})
; => [:tuple [:string {:a :b}] :user/testbaz]
#2021-04-2916:12eoliphantah it was just that? lol. yeah, for now i think just a bit in the docs, maybe in the areas that ref using qualified keys, custom/local registries would be a start just so folks are aware. I just happened to catch Arne’s quick vid on Malli and data modeling, and so gonna just take his approach of my own ‘schema’ name space that limits the surface area, and I can just jack in handling this. will think about what might be a better approach longer term. it does ‘feel’ that something like (m/properties :test vs ::test …) should ‘just work’ from the average user/client’s perspective, returning the same kind of representation. maybe an ‘easy’ 🙂 namespace like reitit?#2021-04-2918:39ikitommineed to think what would be a good resolution for this. today, my 2 cents are to make it work like spec: • by default defer eager references on m/schema, and add an option not to do that (used in form creation) would be a breaking change.#2021-04-2918:41ikitommiunrelated, but, content-dependent collection schemas on coming up. not sure how useful, but possible:
(def List
  (m/-collection-schema
    (fn [properties [child]]
      {:type 'List
       :pred list?
       :empty '()
       :type-properties {:error/message "should be a list"
                         :gen/schema [:vector properties child]
                         :gen/fmap #(or (list* %) '())}})))

(m/validate [List :int] '(1 2))
; => true

(-> (m/explain [List :boolean] [1 2 3])
    (me/humanize))
; => ["should be a list"]

(mg/sample [List {:gen/min 4, :gen/max 10} :int])
;((-1 0 -1 -1 -1 -1 -1 0)
; (-1 -1 -1 0 -1)
; (0 0 1 -2 -1)
; (-1 0 -1 1 -1 0 -4 -1)
; (-2 0 0 -2 1 0)
; (-2 7 2 3 2 -1)
; (-7 -3 -1 7 -1)
; (0 -12 0 8)
; (-5 25 0 -2 -5 1 -1 -25 -10 1)
; (53 221 0 15 1 -42))

(m/form [List :int])
; => [List :int]

(mu/assoc [List :int] 0 :boolean)
; => [List :boolean]
#2021-04-3001:46pserranoHello. I'm looking forward to use malli in a project and will be actively learning about it in the next days 🙂#2021-04-3011:15ikitommihiccup:
[:schema
 {:registry
  {"Order" [:map
            [:items [:vector [:enum "SIM" "SAM"]]]
            [:delivery [:enum "letter" "email"]]]
   "SimDeliveryRule" [:fn {:error/message "If the order only includes a SIM card then the delivery method may be "letter"."
                           :error/path [:delivery]}
                      '(fn [{:keys [items delivery]}]
                         (or (not= items ["SIM"])
                             (= delivery "letter")))]}}
 [:and "Order" "SimDeliveryRule"]]
maps (maybe compact syntax via schema parsing):
{:type :schema
 :registry {"Order" {:type :map
                     :entries [[:items {:type :vector
                                        :items {:type :enum
                                                :values ["SIM" "SAM"]}}]
                               [:delivery {:type :enum
                                           :values ["letter" "email"]}]]}
            "SimDeliveryRule" {:type :fn
                               :error/message "If the order only includes a SIM card then the delivery method may be "letter"."
                               :error/path [:delivery]
                               :value '(fn [{:keys [items delivery]}]
                                         (or (not= items ["SIM"])
                                             (= delivery "letter")))}}
 :value {:type :and
         :values ["Order" "SimDeliveryRule"]}}
#2021-04-3012:00Yehonathan SharvitHello there! Malli is great. A question regarding function validation (a.k.a => ): Is there a way to automatically validate that all function calls are made with proper args during development time?#2021-04-3012:58borkdude@viebel More info here: https://clojurians.slack.com/archives/CLDK6MFMK/p1619006445487300#2021-04-3013:18Yehonathan SharvitThank you @borkdude!#2021-04-3013:19Yehonathan SharvitGiven that instrumentation is not yet there in malli, what’s the common use case for :=>?#2021-04-3013:19Yehonathan Sharvit“Only” generative testing?#2021-04-3013:20borkdudeIt's a good question, I've also wondered myself.#2021-04-3013:20borkdudeAt least you can generate some clj-kondo type hints with it =)#2021-04-3013:22Yehonathan SharvitI just tried clj-kondo type hints and it’s awesome. A question related to that: where is the appropriate place for putting the (mc/emit!) function call?#2021-04-3013:22borkdudeMaybe in "component system du jour" start/reset?#2021-04-3013:23Yehonathan SharvitI didn’t know you were speaking french. I don’t have any component: I am writing a library#2021-04-3013:24Yehonathan SharvitI am also wondering how the emitted config from the library is going to be integrated in the application that uses the lib#2021-04-3013:24borkdudeyou probably shouldn't do this in your library, it's something an end user should do#2021-04-3013:25borkdudeMaybe emit it as part of a pre-commit hook or something#2021-04-3013:25Yehonathan Sharvitwhy wouldn’t my library be responsible for emitting clj-kondo config for the function it provides?#2021-04-3013:26borkdudeYou can do that, but you should just commit that config into git and not generate it at runtime#2021-04-3013:26borkdudeclj-kondo has a mechanism to pick up on configs from libraries#2021-04-3013:27borkdudeDescribed here: https://github.com/clj-kondo/clj-kondo/blob/master/doc/config.md#exporting-and-importing-configuration#2021-04-3013:30Yehonathan SharvitNice!#2021-04-3013:51Yehonathan SharvitIs there already a script that generates clj-kondo config from project that uses malli (basically that calls (mc/emit!)?#2021-04-3013:51Yehonathan SharvitNot sure I could use babashka for that as my library might not be babashka compatible#2021-04-3013:52borkdudeMalli is not babashka compatible either, so you should use a JVM script / function for this. You could just write a function which you can call with clojure -X my.lib/gen-clj-kondo#2021-04-3013:52Yehonathan SharvitMalli is not babashka compatible. That is a sacrilege!#2021-04-3013:53borkdudeThere is an issue / petition to add malli to bb: https://github.com/babashka/babashka/issues/737 Feel free to upvote.#2021-04-3014:01Yehonathan SharvitIt’s not an easy choice: What’s your opinion on that? Did you upvoted or downvoted @borkdude?#2021-04-3014:05Yehonathan SharvitAnother question related to malli’s :=> : Is there a way to automatically generate a doc string?#2021-04-3014:42ikitommi@viebel function instrumentation comes with this: https://github.com/metosin/malli/issues/349#2021-04-3014:44ikitommilot of things are still TODO, help welcome#2021-04-3014:55Yehonathan SharvitYeah. @borkdude pointed me to this Github issue.#2021-04-3014:55Yehonathan SharvitWe are starting to embrace malli at work. Hopefully, we’ll be motivated to help#2021-04-3014:56Yehonathan SharvitA question related to maps. Is it possible to specify that some combination of fields are forbidden?#2021-04-3014:56borkdude@viebel What made you choose malli over spec or both?#2021-04-3014:57Yehonathan Sharvit@ikitommi gave us a great talk about Malli at our dev meetup at work 😜#2021-04-3015:00Yehonathan SharvitWe chose malli mostly because: 1. In malli, schemas are data (not macro required) 2. spec seems stuck. Not sure if spec2 will be compatible with spec 1 3. It’s easier to contribute to malli than to spec #2021-04-3015:05borkdudeInteresting, thanks for sharing. It seems there are two ways people are choosing libraries: 1) choose core unless ... , because: bundled with clojure (no additional deps), authority (core, cognitect), promoted as "the default" 2) choose community, easier to contribute / freedom, usually more focus/options than core libs#2021-04-3015:14Yehonathan SharvitDon’t forget the data driven aspect of malli!#2021-04-3015:23mynomotoHaving a version of spec alpha with a replacement in development for years doesn't exactly inspire confidence.#2021-04-3015:23borkdude@viebel In the defn podcast you argued that namespaced (fully qualified) keywords are a vital part of data oriented programming. spec promotes this by having specs bound to global keywords so specs describe meaning without context. Is this aspect sufficiently present in malli?#2021-04-3015:25borkdude(btw, I liked the podcast :))#2021-04-3015:29Yehonathan SharvitGood question @borkdude. I need to play more with Malli in order to answer this question. (I am so glad you liked it)#2021-04-3015:39Yehonathan SharvitIs there a way to spec a map using a key from a registry but having the key non namespace qualified?#2021-04-3016:53ikitommi
[:map {:registry {::id :int}
 [:id ::id]]
#2021-04-3017:59borkdude#2021-04-3017:59borkdudeThe work Jim is doing is maybe interesting for the authors of malli as well#2021-05-0215:03ikitommidefinitely, looks great @jimka.issy. Malli’s primary goal has been to be a data-driven runtime schema engine, but currently bending it for development time tooling to see how far we can go. Some wishes: • a schema-base case-macro, with clj-kondo based error reporting on non-exhaustive matches - for things like :or and :multi. Genus seems to have those already? • schematized fns. so many syntax options, the lisp curse? • as soon as someone builds a pluggable intellisense / code completer for clojure, happy to emit malli-mappings so that one could complete known map keys (and get errors on invalid keys - via clj-kondo).#2021-05-0215:29ikitommicurrent status: https://twitter.com/ikitommi/status/1363753100268421122#2021-05-0215:03ikitommidefinitely, looks great @jimka.issy. Malli’s primary goal has been to be a data-driven runtime schema engine, but currently bending it for development time tooling to see how far we can go. Some wishes: • a schema-base case-macro, with clj-kondo based error reporting on non-exhaustive matches - for things like :or and :multi. Genus seems to have those already? • schematized fns. so many syntax options, the lisp curse? • as soon as someone builds a pluggable intellisense / code completer for clojure, happy to emit malli-mappings so that one could complete known map keys (and get errors on invalid keys - via clj-kondo).#2021-05-0100:58Alexis VincentAre there any projects or examples out there for nifty crux/Malli compat stuff. Schemas or crux helper functions etc#2021-05-0110:17refsetGood timing! See @U0A5V8ZR6 's new lib https://mobile.twitter.com/spacegangster/status/1388092289084428299#2021-05-0110:22Ivan Fedorov@U051V5LLP is the initiator and the sponsor on this one 🙏#2021-05-0111:17Alexis VincentYeah so cool! I had a 5 min play with this yesterday! Great work!#2021-05-0111:18Alexis VincentAny further examples/projects anyone has seen? @U899JBRPF#2021-05-0112:12refsetNothing else that's hit my radar yet, but I'm also very interested by the intersection :) Semi-related: I am also curious about the possibility of Malli->Alloy tooling that could assist with modelling in Crux#2021-05-0113:12Alexis VincentCan connect in a bit and see what we come up with#2021-05-0114:20dvingo@UJWLUPW13 do you have any use-cases in mind you're looking for? or dev workflows? I have some of my own ideas for helper code I'd want related to crux, but I'm interested to hear of others.#2021-05-0114:21dvingo@U899JBRPF do you have any links or materials on alloy as it relates to crux?#2021-05-0114:32Alexis VincentNot yet. Just getting started integrating malli and crux (and new to both). Just wrote a transformer that santises crux entities to be passed on to a json api. transforms :crux.db/id -> id and strips out entity type im transacting in. Will get a better feel for what I need as I work with both#2021-05-0114:36Alexis VincentMalli coersion for crux database migration is going to be 👌#2021-05-0114:36Alexis VincentAnd UI generation#2021-05-0215:15Ivan FedorovMr Reiman, maybe you could be interested making a video review of the EQL generation I wrote and giving your opinion on a better way to write schema transformers? I think this could be beneficial for those writing transformers based on malli. If yes — we could schedule a video call @U055NJ5CC If no — no worries.#2021-05-0305:23ikitommisure @U0A5V8ZR6, would like to do that, just super busy at work atm, between projects. If the code - and the goal - is available somewhere, I could first try to read that, between things.#2021-05-0308:19Ivan FedorovYep, you can look at it here: https://github.com/dvingo/malli-code-gen/blob/main/src/malli_code_gen/gen_eql.clj We have a branch merge pending. If we merge it before you look at the code – we’ll leave you a link in https://github.com/dvingo/malli-code-gen cc @U055NJ5CC Not rushing you with the review. If you can say at what period you will be probably available for that (in a week or three) that would help. Good luck with the workload!#2021-05-0318:02refsetHey @U051V5LLP > do you have any links or materials on alloy as it relates to crux? Nothing published from me yet, but this is worth a look https://www.hillelwayne.com/post/formally-modeling-migrations/ and, whilst not directly Alloy-related, I think principles of https://en.wikipedia.org/wiki/Object-role_modeling are also relevant#2021-05-0323:27dvingoooohh this looks great, thanks for sharing! I have some reading to do 🙂#2021-05-0115:08Joeli'm mucking with jsonista/malli and would like to use keywords for some map values, eg: { :Nested { :Map { :enums "values" :other "string" }}} How do i get the values for :enums to be keywords, eg :values... yet leave other strings as strings?#2021-05-0118:14ikitommi@joel380 you need to describe a schema with all the parts that need transformations. in your case, simplest way to do it would be:
(require '[malli.provider :as mp]
         '[malli.core :as m]
         '[malli.transform :as mt])

;; infer schema from example value
(def schema (mp/provide [{:Nested {:Map {:enums :values}}}]))

;; create json-decoder for the schema
(def from-json (m/decoder schema (mt/json-transformer)))

;; apply
(from-json {:Nested {:Map {:enums "values" :other "string"}}})
; => {:Nested {:Map {:enums :values, :other "string"}}}
#2021-05-0203:59JoelThanks I got that working. I had a schema with [:enum :value1 :value2] I assumed that would do the conversion, but it didn't work until I added [:and keyword? [:enum ...]]#2021-05-0204:00Joelmakes sense though.
#2021-05-0204:02JoelHow can I validate the length of strings?#2021-05-0206:08ikitommi[:string {:min 1, :max 10}]#2021-05-0218:44Joel@ikitommi How do i see the list of keywords I can specify? Is it to look at unit tests? For example, I didn't know where to find the min/max keywords.#2021-05-0305:25ikitommi@joel380 not atm. The new malli schemas of malli schemas feature allows us to describe the available properties formally as schemas, but it’s just the mechanism atm, no content shipped. Once all schmas are described (both properties and children), it will be THE documentation. For now, you need to read the sources. Documentation PRs always welcome.#2021-05-0305:29ikitommie.g.
;; currently
(m/properties-schema :string)
; => nil

;; after
(m/properties-schema :string)
; [:map
;  [:min {:description "length must be at least"} :int]
;  [:max {:description "length must be at most"} :int]]
… different schema applications (e.g. generation) will have their own overlays, which can be merged to get the full set of available keys. Should be easy to generate proper docs out of those.
#2021-05-0305:30ikitommiafter the children & property schemas are done, one can also generate valid schema-hiccup out of a given registry 🙂#2021-05-0312:45Yehonathan SharvitIs there a way to forbid some combination of keys in a map? For instance a map with either :a and :b or :c and :d#2021-05-0312:48borkdudeprobably using another predicate?#2021-05-0312:48borkdudeor using two different maps#2021-05-0312:51Yehonathan SharvitWhat do you mean by “using another predicate”?#2021-05-0312:53borkdude[:and [:map ...] [:fn (fn [...] ...)]]#2021-05-0312:54borkdudeSame as with clojure.spec#2021-05-0312:54borkdude
(def my-schema
  [:and
   [:map
    [:x int?]
    [:y int?]]
   [:fn (fn [{:keys [x y]}] (> x y))]])
#2021-05-0312:55Ben SlessI was just going to ask if there's a way to specify constraints on relations between keys which does not involve defining an ad-hoc function#2021-05-0312:56borkdudeMaybe using two separate closed maps also works. I think it depends on your domain perhaps?#2021-05-0312:57Ben SlessLikely, although I was just thinking of a case like the one you illustrated, a constraint where the value at one key is greater than another#2021-05-0312:59Ben SlessThis pattern repeats so often, should it be a schema?#2021-05-0313:19Ben SlessSomething like:
(def preds {:> >, :>= >=, :< <, :<= <=, := =, :not= not=})

(defn -rel
  []
  (m/-simple-schema
   (fn [_ [pred a b]]
     {:type :rel,
      :pred (m/-safe-pred #((preds pred) (get % a) (get % b))),
      :min 3,
      :max 3})))

(m/validate
 (m/schema
  [:and
   [:map
    [:x :int]
    [:y :int]]
   [:rel :< :x :y]]
  {:registry
   (mr/composite-registry
    m/default-registry
    {:rel (-rel)})})
 #_{:x 1 :y 2}
 {:x 1 :y 1})
#2021-05-0313:23borkdudereminds me of jet :)
$ echo  '{:a 1 :b 2}' | jet --query '(< :a :b)'
true
$ echo  '{:a 1 :b 2}' | jet --query '(> :a :b)'
false
#2021-05-0313:28ikitommiI think it’s good idea to experiment new relational schemas in the user space, both -simple-schema & -collection-schma are good ways to make these, like @ben.sless you demoed. The more stuff pushed into these data-languages, the more it looks like a simple sci.#2021-05-0313:29Ben SlessI also had some thoughts on the subject here https://clojureverse.org/t/declarative-rules-for-relations-between-inputs/7623/5?u=bsless#2021-05-0313:30Ben SlessA big plus it could have over sci is getting the benefit of the JIT while still keeping the schema serializable#2021-05-0313:30ikitommionly 👍#2021-05-0313:31ikitommitrue, sci introduces a lot of overhead. and is big on cljs.#2021-05-0313:31borkdudemuch smaller than self-hosted CLJS though#2021-05-0313:32borkdudebut how many people are really using schema serialization?#2021-05-0313:32borkdudeI think most people are not#2021-05-0313:32ikitommii think so too.#2021-05-0313:32Ben SlessDoes select-keys translate to mu/get of all the keys then closing the schema?#2021-05-0313:33ikitommibut a lightweight map key dependency utility might be a good fit for many things. just (somone) needs (to invent) more syntax.#2021-05-0313:33Ben SlessThe schema serialization isn't my first (or third) priority here tbh, it's more about having more expressive schemas for the common 90% of cases and not having to define ad-hoc functions and errors for them, not to mention generators.#2021-05-0313:34ikitommihttps://github.com/metosin/malli/blob/master/src/malli/util.cljc#L233-L239#2021-05-0313:34ikitommihttps://github.com/metosin/malli/blob/master/src/malli/util.cljc#L233-L239#2021-05-0313:34borkdudeyeah, I think that's nice. I think clojure spec also doesn't have a good answer to this yet and it's pretty common#2021-05-0313:45Yehonathan SharvitIs there an answer to this in TypeScript?#2021-05-0313:49ikitommiI think you have to use union types with ts#2021-05-0314:11Yehonathan SharvitDoes it help to define constraints on map keys#2021-05-0314:11Yehonathan Sharvit?#2021-05-0317:39JoelI'm surprised that TypeScript is sophisticated enough to be used as inspiration, I guess I need to learn more. I've presumed Elm/Haskell are more sophisticated, eg.: https://www.youtube.com/watch?v=IcgmSRJHu_8#2021-05-1002:48mynomotoTypescript is way more pragmatic than elm/Haskell. Structural typing almost looks like malli schemas enforced at compile time.#2021-05-0313:29ikitommithere is also things like :select-keys -> https://malli.io/?value=%7B%3Aa%201%2C%20%3Ab%202%7D&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22ABCD%22%20%5B%3Amap%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aa%20%3Aany%5D%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Ab%20%3Aany%5D%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Ac%20%3Aany%5D%20%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Ad%20%3Aany%5D%5D%7D%7D%20%20%20%20%20%20%20%20%20%20%20%20%20%0A%20%5B%3Aor%20%0A%20%20%5B%3Aselect-keys%20%22ABCD%22%20%5B%3Aa%20%3Ab%5D%5D%0A%20%20%5B%3Aselect-keys%20%22ABCD%22%20%5B%3Ac%20%3Ad%5D%5D%5D%5D.#2021-05-0313:31ikitommiwould meander be great at describing the relations, as data?#2021-05-0313:36borkdudemalleander!#2021-05-0313:37ikitommiusing modified ben’s syntax:
[:and
 [:map
  [:a :any]
  [:b :any]
  [:c :any]
  [:d :any]]
 [:rel
  [:or
   [:and :a :b]
   [:and :c :d]]]]
#2021-05-0313:38Ben SlessThe only objection I can come up with is that meander is too powerful, (i.e. the result will be hard to work with)#2021-05-0313:39Ben Sless"Here, have this fighter-jet fueled by liquid plutonium" Scary#2021-05-0313:38borkdudeI think it can be confusing to overload the meaning of :or within :rel#2021-05-0313:39borkdudeMaybe just [:and [:map ....] [:or [:required-keys :a :b] [:required-keys :c :d]] [:gtk :a :b]]#2021-05-0313:39borkdude(gtk = greather than for keys, or something)#2021-05-0313:40Ben SlessThe rel syntax needs to be properly thought out, I just invented something ad-hoc to see if it could work#2021-05-0313:40borkdude[:keys/> :a :b]#2021-05-0313:40Ben SlessHow about :constraints?#2021-05-0313:40borkdude[:keys/require :a :b]#2021-05-0313:41Ben SlessThen constraints unify by default (like datalog) or disjoin when specified (via :or)#2021-05-0313:41borkdude[:k> :a :b], [:kreq :a :b]#2021-05-0313:42ikitommithis works already:
(m/validate
  [:schema {:registry {::a :int
                       ::b :int
                       ::c :int
                       ::d :int}}
   [:or
    [:map ::a ::b]
    [:map ::c ::d]]] 
  {::a 1, ::b 2})
; => true
#2021-05-0313:49Yehonathan SharvitBut it doesn’t scale. When you have other set of constraints you’d need to write down all the combinations of valid keys.#2021-05-0313:42borkdudeyeah, that's what I said in the start of the conversation: two disjunct map defs#2021-05-0313:42ikitommijust not with non-qualified keys (for no good reason)#2021-05-0313:42Ben Sless
[:constraints
 [:or
  [:and
   [:requires :a :b]
   [:> :a :b]]
  [:and
   [:requires :c :d]
   [:< :c :d]]]]
#2021-05-0313:42borkdudebut for rels between keys you could just have specialized ops like [:k> :a :b]: the :a key must be greater than the :b key, without introducing some new concept#2021-05-0313:43ikitommii kinda like it.#2021-05-0313:44Ben Slessspecial ops rub me the wrong way, for some reason. What's wrong with just :>?#2021-05-0313:44ikitommibut question is: from whom the declaration are for? schema writer? user? external docs?#2021-05-0313:45ikitommi
(m/validate [:> 6] 4)
; => false
#2021-05-0313:45ikitommiit already exists#2021-05-0313:46Ben Slesshm, yes, but as a schema not an argument (if we go back to the [:rel x y z] or [:constraint ,,,] suggestion#2021-05-0313:47borkdude> What's wrong with just :> That it means different things in different contexts, this can be confusing imo.#2021-05-0313:48borkdudeWhat if you want to really do numeric comparison like [:> :a 5] and 5 is also a key in a map?#2021-05-0313:49borkdudeIn jet I chose [:> :a #jet/lit 5]#2021-05-0313:49borkdudebut quickly this became tedious so I wrote a clojure interpreter#2021-05-0313:50Ben SlessThe most shorthand constraint syntax I can think of would be :! An alternative which conforms to your suggestion would be :!/>#2021-05-0313:51Ben SlessDownside - it looks like Perl Upside - concise and unique syntax#2021-05-0313:52borkdudeyou should always make a good trade-off between adding extra syntax (= complexity) and how much people are really going to use this#2021-05-0313:53ikitommi
[:and
 [:map
  [:a :any]
  [:b :any]
  [:c :any]
  [:d :any]]
 [:keys/or
  [:keys/and :a :b]
  [:keys/and :c :d]]]
#2021-05-0313:53borkdudeyeah, I like that a lot better#2021-05-0313:54borkdudewhy even :keys/or, you can just use the normal :or here?#2021-05-0313:55ikitommitrue#2021-05-0313:54Yehonathan SharvitI was about to write the same thing#2021-05-0313:54Ben SlessThen adding a predicate function on the keys would look like
[:and
 [:map
  [:a :any]
  [:b :any]
  [:c :any]
  [:d :any]]
 [:or
  [:and
   [:keys/and :a :b]
   [:keys/< :a :b]]
  [:keys/and :c :d]]]
#2021-05-0313:54Yehonathan SharvitThe disjunction must not be about the keys#2021-05-0313:55Ben Sless:keys/or should mean "at least k1 or k2 should exist"#2021-05-0313:55borkdudewhat does :keys/and mean: both keys must be present? what does [:keys/and :a] mean then? Maybe [:keys/req :a] is a better name?#2021-05-0313:56Ben SlessMaybe even without a shorthand, keys/require#2021-05-0313:57Yehonathan Sharvit:keys/req doesn’t convey the fact that :a and :b are related#2021-05-0313:57borkdudebut what exactly does [:keys/and :a :b] mean then, how are these things related?#2021-05-0313:58Yehonathan SharvitI didn’t say I liked :keys/and#2021-05-0313:59Yehonathan SharvitMaybe [:keys/present :a :b]?#2021-05-0314:00borkdudebut you also didn't say what the semantic relationship between those keys is according to that op. > :keys/req doesn’t convey the fact that :a and :b are related :present also doesn't communicate a relationship, just as :required doesn't, it just means all those keys must be required/present#2021-05-0314:11Yehonathan SharvitI was wrong. There is no relationship between the keys. But somehow I find it weird to say “required” inside an or condition#2021-05-0313:56Yehonathan SharvitAnd now, where would we put a custom predicate [:fn …]?#2021-05-0313:57borkdudeor :keys/with and :keys/without ;P#2021-05-0313:57ikitommioff to cook some food. good discussion, as always 👍#2021-05-0313:58Ben SlessThank you, feel like we hit on something useful and needed Bon appetit!#2021-05-0314:00Yehonathan SharvitBy the way, my use case is real for a project at work.#2021-05-0314:00Ben Slesssame#2021-05-0314:00Yehonathan SharvitWe are writing a Hbase driver#2021-05-0314:00borkdudeI guess you can already express required keys like this as well? [:and [:map ...] :a :b :c] or do keywords not behave like predicates in malli?#2021-05-0314:01Ben Sless
(m/schema [:and :a :b])
exception
#2021-05-0314:02borkdudeok, [:and [:fn :a] [:fn :b]] ? :thinking_face:#2021-05-0314:02borkdude(ugly)#2021-05-0314:03Ben SlessAnd non informative, and useless for generation
(me/humanize (m/explain (m/schema [:and [:fn :a]]) {:b 1}))
;; => #:malli{:error ["unknown error"]}
#2021-05-0314:03borkdudeagreed#2021-05-0314:01Yehonathan SharvitThe scan function receives a map that could contain either :from and :to or :starts-with#2021-05-0314:01borkdudeIt looks like malli is becoming really popular in Israel?#2021-05-0314:02Yehonathan SharvitThere are not so many Clojure shops i Israel. Therefore, we could already say that 10% of Israeli Clojure shops use malli 😁#2021-05-0314:06Yehonathan SharvitI’m struggling to write the schema of the map received by scan
[:map    
   [:starts-with :string]
   [:from :string]
   [:to :string]
 [:xor 
   [:keys/req :starts-with]
   [:keys/req :from :to]
   [:keys/req :from]
   [:keys/req :to]
 ]
]
#2021-05-0314:06Yehonathan Sharvit:or is not good in my case. It should be exclusive or#2021-05-0314:07borkdudeif it is xor, then why not use disjunct maps?#2021-05-0314:07Yehonathan SharvitBecause there are other keys#2021-05-0314:07borkdudemaybe add a :type field or something?#2021-05-0314:07Yehonathan Sharvit:time-range , :limit#2021-05-0314:08Yehonathan Sharvit
[:map    
   [:starts-with :string]
   [:from :string]
   [:to :string]
   [:limit :number]
   [:time-range [:map [:from-ms :int] [:to-ms :int]]
 [:xor 
   [:keys/req :starts-with]
   [:keys/req :from :to]
   [:keys/req :from]
   [:keys/req :to]
 ]
]
#2021-05-0314:08Yehonathan SharvitWhat is a :type field?#2021-05-0314:09borkdudeI think you could have different types of ranges#2021-05-0314:11borkdude{:range-type :closed}, {:range-type :open}#2021-05-0314:11Ben SlessIn this case I'd start with some base schema, merge the different options with it and or between them#2021-05-0314:12Yehonathan SharvitI don’t get what both of you suggested#2021-05-0314:17borkdude@viebel I think this requirement is a bit weird:
:xor
[:keys/req :from :to]
[:keys/req :from]
if from is present, and to is not, then the second is true. but if to is present, then the first one is true. so to is optional.
#2021-05-0314:17borkdudeI did not look at the other requirements, but I think your logic can be "refactored"#2021-05-0314:17borkdudeby modeling it differently#2021-05-0315:28Yehonathan SharvitI don’t see how to refactor the logic. All the followings are valid • :from 10 :to 20 • :form 10 • :to 20 • :starts-with aaa#2021-05-0315:28Yehonathan SharvitThe followings are invalid#2021-05-0315:29Yehonathan Sharvit• :from 10 :starts-with aaa • :to 20 :starts-with aaa • “from 10 :to 20 starts-with aaa#2021-05-0314:22emccuehmm, this is a tricky one#2021-05-0314:23emccue
[:or [:map {:closed true}
       [:from ...]
       [:to   ...]]
     [:map {:closed true}
       [:starts-with ...]]]
#2021-05-0314:24emccueOR is an exclusive or if the types don't intersect#2021-05-0314:24emccue
[:map 
  [:from ...]
  [:to   ...]]
#2021-05-0314:24emccuebut if you have it closed you can't have anything other than from and to#2021-05-0314:25emccueand if you have it open you leave open the possibility that :starts-with is in the map as well#2021-05-0314:25emccuewhat you need is basically#2021-05-0314:27emccue
[:or [:and [:map 
             [:from ...]
             [:to ...]]
           [:not [:map 
                   [:starts-with ...]]]]
     [:and [:map 
             [:starts-with ...]]
           [:not [:map 
                   [:from ...]
                   [:to   ...]]]]
#2021-05-0314:27emccueOR#2021-05-0314:27emccue
[:or [:and [:map 
             [:from ...]
             [:to ...]]
           [:not [:map 
                   [:starts-with ...]]]]
     [:and [:map 
             [:starts-with ...]]
           [:not [:map [:from ...]]
           [:not [:map [:to ...]]]]]
#2021-05-0314:28emccuenot that not exists, but conceptually thats what you would need to represent the constraint#2021-05-0314:28emccuesince you only want to be closed against specific keys#2021-05-0314:29emccueif you are willing to explicitly enumerate all the keys you can just do the version with {:closed#2021-05-0314:29Ben SlessI meant something like
(def Base
  [:map
   [:limit :number]
   [:time-range [:map [:from-ms :int] [:to-ms :int]]]])

(def S1 (mu/merge Base [:map [:starts-with :string]]))
(def S2 (mu/merge Base [:map [:from :string] [:to :string]]))
#2021-05-0315:02Ben SlessDid some generalizing work, still not set on the syntax
(defn comparator-relation
  [sym msg]
  (let [f @(resolve sym)
        type (keyword "!" (name sym))]
    [type
     (m/-simple-schema
      (fn [_ [a b]]
        (let [fa #(get % a)
              fb #(get % b)]
          {:type type
           :pred (m/-safe-pred #(f (fa %) (fb %))),
           :type-properties
           {:error/fn
            {:en (fn [{:keys [schema value]} _]
                   (str
                    "value at key "
                    a ", "
                    (fa value)
                    ", should be "
                    msg
                    " value at key "
                    b
                    ", "
                    (fb value)))}}
           :min 2,
           :max 2})))]))

(defn -comparator-relation-schemas
  []
  (into
   {}
   (map (fn [[sym msg]] (comparator-relation sym msg)))
   [['>  "greater than"]
    ['>= "greater than or equal to"]
    ['= "equal to"]
    ['== "equal to"]
    ['<= "lesser than or equal to"]
    ['< "lesser than"]]))


(me/humanize
 (m/explain
  (m/schema
   [:and
    [:map
     [:x :int]
     [:y :int]]
    [:!/> :x :y]]
   {:registry
    (mr/composite-registry
     m/default-registry
     (-comparator-relation-schemas))})
  {:x 1 :y 1}))
#2021-05-0406:24Ben SlessNext step is figuring out how to derive generators from this#2021-05-0407:14Ben SlessMost cases just work besides equality#2021-05-0408:28Ben SlessHere we go:
(defn- derive-from-fmap
  [schema options gen]
  (let [props (merge (m/type-properties schema)
                     (m/properties schema))]
    (when-some [fmap (:gen/fmap props)]
      (gen/fmap (m/eval fmap (or options (m/options schema)))
                gen))))

(defmethod mg/-schema-generator :and [schema options]
  (let [[h & t] (m/children schema options)
        base-gen (mg/generator h options)
        gen (reduce (fn [gen schema]
                      (if-let [gen (derive-from-fmap schema options gen)]
                        gen
                        gen))
                    base-gen
                    t)]
    (gen/such-that (m/validator schema options) gen 100)))
#2021-05-0410:59Ben SlessI collected the proof of concept implementation with some explanations and cleaned up code at a repo https://github.com/bsless/malli-keys-relations#2021-05-0412:53Ben SlessSome more thoughts: • should references to paths inside maps be explicit rather than implicit? i.e. [:keys/> :x :y] vs [:keys/> [:path :x] [:path :y]] . The second syntax is more cumbersome but gives more freedom (`[:path :x :z]`) and facilitates more logic (see next points) • Facts about collections. How do I say the value at key k1 must be contained in the collection in k2 , or a number smaller than the size of that collection? Stuff like [:contains? :x :y] (the value at y is in x), or [:> [:count :x] :y] (y is smaller than the count of x)#2021-05-0413:46dvingoVery cool stuff! I was playing with adding support for cljs yesterday. (resolve sym)(https://github.com/bsless/malli-keys-relations/blob/master/src/com/github/bsless/malli_keys_relations.clj#L112) is not allowed in cljs - I suspect converting this to a macro may get cljs support working. In cljs resolve only allows a literal quoted symbol as an argument: https://cljs.github.io/api/cljs.core/resolve#2021-05-0414:07borkdudeNote that runtime resolve in general doesn't play well with GraalVM native-image unless it's executed at compile time#2021-05-0414:20Ben SlessI'll replace it with a map or defmulti#2021-05-0414:20Ben SlessI think defmulti will be best, extensible#2021-05-0415:28Ben Slessdone#2021-05-0415:29Ben SlessWhat about the alternative syntax?#2021-05-0506:48ikitommigreat work @ben.sless! need to read the code & think about this a bit. could you check the corresponding features from JSON Schema (https://json-schema.org/understanding-json-schema/reference/object.html#dependencies) so that Malli can be as compliant as it’s reasonable with it. I think your suggestion already a superset of what is in JSON Schema…#2021-05-0506:49ikitommimeanwhile, a quick poke on the schema inferring / providing: • part1: make it suck less (5x faster on sample dataset): https://github.com/metosin/malli/pull/439 • part2: make inferring first class, most likely 100x faster: https://github.com/metosin/malli/pull/440#2021-05-0506:52ikitommiwith part2, all schemas are reponsible for describing how values can/not be described with schemas. This will open up things like inferring types from enums etc, e.g. [:enum "small" "medium" "large"] has a child schema of :string.#2021-05-0506:53ikitommithe part1 (5x faster) of new provider is now 68 loc, instead of the old 69. The old was kinda bloated anyways 😉#2021-05-0506:56ikitommi
➜  malli git:(faster-inferrer-part1) ✗ cloc src --by-file
      15 text files.
      15 unique files.
       0 files ignored.

 v 1.86  T=0.03 s (571.3 files/s, 180498.1 lines/s)
----------------------------------------------------------------------------------------
File                                      blank        comment           code
----------------------------------------------------------------------------------------
src/malli/core.cljc                         146             46           1779
src/malli/impl/regex.cljc                    93             23            508
src/malli/util.cljc                          44             15            350
src/malli/transform.cljc                     64             22            335
src/malli/generator.cljc                     48             13            281
src/malli/error.cljc                         24              6            249
src/malli/clj_kondo.cljc                     21              3            141
src/malli/json_schema.cljc                   23              3            128
src/malli/dot.cljc                            7              3             69
src/malli/provider.cljc                      10              3             68
src/malli/registry.cljc                      15              3             65
src/malli/swagger.cljc                       14              4             51
src/malli/impl/util.cljc                      7              0             24
src/malli/edn.cljc                            3              0             16
src/malli/sci.cljc                            1              0             11
----------------------------------------------------------------------------------------
SUM:                                        520            144           4075
----------------------------------------------------------------------------------------
#2021-05-0506:57ikitommi
➜  malli git:(faster-inferrer-part1) cloc test --by-file
      11 text files.
      11 unique files.
       0 files ignored.

 v 1.86  T=0.04 s (264.8 files/s, 133395.9 lines/s)
-----------------------------------------------------------------------------------
File                                            blank        comment           code
-----------------------------------------------------------------------------------
test/malli/core_test.cljc                         330              6           1902
test/malli/util_test.cljc                         115              4            789
test/malli/transform_test.cljc                     93              3            714
test/malli/error_test.cljc                         42              3            439
test/malli/generator_test.cljc                     39              3            266
test/malli/swagger_test.cljc                       11              4            258
test/malli/json_schema_test.cljc                   14              4            236
test/malli/dot_test.cljc                           12              0             73
test/malli/registry_test.cljc                      10              2             61
test/malli/clj_kondo_test.cljc                      8              0             55
test/malli/provider_test.cljc                       4              0             41
-----------------------------------------------------------------------------------
SUM:                                              678             29           4834
-----------------------------------------------------------------------------------
#2021-05-0507:26Ben SlessRegarding part 1, from the profiling I did yesterday it looks like most CPU is based on miu/-fail. I'll profile again with part1 merged to see what conclusions I can infer#2021-05-0508:14Ben Slessdisregard the last message I had some repl state#2021-05-0508:16Ben Slessok, see ~2x speedup#2021-05-0508:18Ben Sless-fail still dominates CPU, though. Would you like me to prepare a MR for the exception speedup?#2021-05-0518:22Ben SlessQuestion about Malli's design - what's the rationale behind IntoSchemas? Why aren't Schemas and local bindings sufficient?#2021-05-0521:31ikitommi> For internal elegance, Malli is built using protocols. > It,s much easier to program and reason about the system with protocols than against functions / data. IntoSchema is analogous to a Class and Schena to an Object. There is one shared instance of :map IntoSchema (in a registry) which describes how to create :map Schema instances. And soon how to infer schemas from values. Kinda like adding static methods on classes, except it's all polymorphic.#2021-05-0612:19mike_ananevwhat is the difference between :or and :alt ?#2021-05-0612:22mike_ananevfound in docs, :alt is for spec inside seq#2021-05-0612:59mike_ananev@ikitommi how to define string constant in map spec? e.g version api is a constant#2021-05-0613:00mike_ananev[:enum "v1.0.0"] ?#2021-05-0613:00mike_ananevis there other ways?#2021-05-0613:02mike_ananevand why sets are not supported as spec definition? I cannot put
[:abc #{1 2 3}]
in map spec
#2021-05-0613:11ikitommi[:= "v1.0.0"] or [:enum "v1.0.0"], both work.#2021-05-0613:14ikitommi#{1 2 3} doesn’t work as we didn’t want to reserver too much clojure syntax for special purposes. There is a hook to add support for that in the user space, but not documented as it’s not recommended.#2021-05-0613:15ikitommialso, now I think adding a shortcut syntax for regexps was not a good idea.#2021-05-0613:16ikitommiwhy? there is a cljs compiler warning about those, coudn’t fix it, instead of: #"kikka.*" would have been enough to have [:re #"kikka.*] or even [:string {:format "kikka.*"}]#2021-05-0613:18ikitommiI think in 90%+ cases people add anyway properties to the regexps like :error/message, so the benefit for supporting plain regexps is quite small.#2021-05-0613:25Ben SlessAn enhancement of regex error messages could be an indication at which character the match has failed. "Should match regex" isn't terribly helpful for humanized errors#2021-05-1008:49nilernHost regexes don't support error positions. And I don't blame them, because the regex can fail in multiple ways at every character (our seqex schemas heuristically give the error the first longest partial match).#2021-05-1014:59Ben SlessIf we can do regex based generators we should theoretically be able to do regex based error reporting No one said it would be easy, or even a good idea. You may not want to expose your regex via error messages#2021-05-0613:51mike_ananev[:= "v1.0.0"]👍#2021-05-0616:53JoelIs there a function that would give ":and" instead of ":or"? stricter vs. looser. maybe mu/intersect?
(mu/union [:map [:Event keyword?]] [:map [:Event [:enum :A :B]]])
=> [:map [Event [:or keyword? [:enum :A :B]...
#2021-05-0704:01ikitommiyou can configure the the mu/merge with few options, see`:merge-default` in mu/union https://github.com/metosin/malli/blob/master/src/malli/util.cljc#L104-L113#2021-05-0704:03ikitommiWould that be mu/intercect with :and ? If so, PR welcome.#2021-05-0710:49ikitommi(my set theory skills are rusted on fridays)#2021-05-1008:52nilernIntersection would go with :and (which is a lattice meet)#2021-05-0618:50Ivan Fedorovany way to get schema name from RefSchema? e.g. I have [:schema {:registry reg} ::task] and it’s already a RefSchema#2021-05-0619:22Ben Sless(m/deref schema)?#2021-05-0619:22Ben Sless
(let [schema (schema ?schema options)]
     (cond-> schema (satisfies? RefSchema schema) (-deref)))
#2021-05-0619:23Ben SlessLooks like just what you need#2021-05-0619:43Ivan Fedorov@UK0810AQ2 thanks for the input! This gives something that looks like a keyword but in fact is a :malli.core/schema and I don’t understand how to get the keyword out#2021-05-0619:46Ben Slessah, hold on, let's dig some more#2021-05-0619:49Ben SlessWhat's wrong with just calling m/form?#2021-05-0619:53Ben SlessOkay, this is right:
(m/form (m/deref S))
You'll get back a keyword
#2021-05-0619:54Ivan Fedorovwhat you see in m/children is still a schema! but (-> m/form last) cuts it, thanks!#2021-05-0714:49Siddharth Jainis there a way I can set all map schema :closed by default? so that don't need to specify individually.#2021-05-0714:59dharriganI don't think there is, however, you could gather up all your schemas and map over them with malli.utils/closed-schema to recursively close them.#2021-05-0714:59dharriganThat may work#2021-05-0715:06dvingoYou could copy the -map-schema definition and make your own -closed-map-schema https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L616 so you could write: [:closed-map [:xyz string?]] not sure if that's worth it vs just typing {:closed true} though#2021-05-0715:17Siddharth Jainthanks guys, sounds good, cheers#2021-05-0716:10ikitommi@siddharthjain.in or better: create a PR that allows m/-map-schema to take :closed and then say:
(m/-map-schema {:naked-keys true, :closed true})
.. and you have a closed variant.
#2021-05-0815:25cjsauerIs it possible to get malli to coerce blank strings "" into the :default value? Example:
(m/decode
  [:map [:x {:default 0} int?]]
  {:x ""}
  (mt/transformer
    mt/default-value-transformer
    mt/string-transformer))
Should result in {:x 0}
#2021-05-0815:34cjsauerEh, never mind. I’m thinking this isn’t a great idea. I can just strip empty strings from my data before sending it into malli for coercion. I think that’s clearer and less error-prone.#2021-05-0817:53dvingoI noticed that the the schema:
[:schema {:registry {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]}}
 ::cons]
doesn't work on http://malli.io (it doesn't produce DOT output or JSON schema) while
[:schema
 {:registry {"ConsCell" [:maybe [:tuple :int [:ref "ConsCell"]]]}}
 "ConsCell"]
this one does work. In my applications I would like to use fully qualified keyword schemas (RefSchemas) is there a way to get the transformers to work without walking the schema ahead of time and converting all fully qualified keywords to strings?
#2021-05-0818:06ikitommi@danvingo it's a malli-sci-thing. I think one needs to define what is the current ns to sci so that ::cons works. Try :user/cons and it should work. I think @borkdude knows the answer how to make the :: work with sci...#2021-05-0818:14borkdude@ikitommi ::foo resolves to whatever the current namespace is, by default :user/foo#2021-05-0819:16dvingothanks Tommi - using :user/cons worked. I was also seeing some issues in a local repl regarding refschemas and dot. I'll try to get minimal repro#2021-05-0820:16borkdude
Welcome to xterm-sci.
user=> ::foo
:user/foo
user=> 
https://babashka.org/xterm-sci/
#2021-05-0820:16ikitommihmm... Might be a edamame thing?#2021-05-0901:11dvingoOk here's a small sample that I'm perplexed by:
(ns my-app.my-ns
  (:require [malli.core :as m]
            [malli.dot :as dot]))

(def comment-schema
  {::id         :uuid
   ::content    :string
   ::replies    [:vector [:ref ::comment]]
   ::updated-at inst?
   ::created-at inst?
   ::comment    [:map
                  ::id
                  ::content
                  [::replies {:optional true}]
                  ::updated-at
                  ::created-at]})

(def reg {:registry (merge (m/default-schemas) comment-schema)})

(def c
  {::id         #uuid"ff43dad0-8007-47d7-845e-0b20a021bedb"
   ::content    "hello"
   ::created-at (java.time.Instant/now)
   ::updated-at (java.time.Instant/now)})

(m/validate (m/schema ::comment reg) c)
; => true

(m/schema ::comment reg)
; => ::comment
I'm confused why this fails:
(m/schema [:schema reg ::comment] reg)
; =>
; Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:17).
; :malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1}
as well as these calls:
(dot/transform (m/schema ::comment reg))
; =>
; Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:17).
; :malli.core/invalid-schema {:schema ::comment}

;; trace:
                             malli.dot/transform                dot.cljc:   58
                             malli.dot/transform                dot.cljc:   60
                              malli.dot/-collect                dot.cljc:   17
                                 malli.core/walk               core.cljc: 1655
                                 malli.core/walk               core.cljc: 1657
     malli.core/-schema-schema/reify/reify/-walk               core.cljc: 1276
                    malli.core/walk/reify/-outer               core.cljc: 1662
                           malli.dot/-collect/fn                dot.cljc:   21
              malli.core/-properties-and-options               core.cljc:  270
                   malli.core/-property-registry               core.cljc:  265
                          clojure.core/reduce-kv                core.clj: 6856
                     clojure.core.protocols/fn/G           protocols.clj:  175
                                 clojure.core/fn                core.clj: 6845
                                             ...                              
                malli.core/-property-registry/fn               core.cljc:  265
                               malli.core/schema               core.cljc: 1610
                              malli.core/-schema               core.cljc:  251
                          malli.impl.util/-fail!               util.cljc:   16

(dot/transform (m/schema ::comment reg) reg)
; =>
; Execution error (ExceptionInfo) at malli.impl.util/-fail! (util.cljc:17).
; :malli.core/invalid-schema {:schema ::comment}
;; same trace as above
#2021-05-0902:22dvingoThis one also fails..
(def cons-schema {::cons [:maybe [:tuple pos-int? [:ref ::cons]]]})
(def cons-list [1 [2 [3 [4 [5 nil]]]]])
(def reg {:registry (merge (m/default-schemas) cons-schema)})
(dot/transform (m/schema ::cons reg) reg)
; => 
; :malli.core/invalid-schema {:schema ::cons}
#2021-05-0902:22dvingobut this works: (m/validate (m/schema ::cons reg) cons-list) => true#2021-05-0922:59dvingoFollowing up on this, made some further investigations. this works:
(dot/transform [:schema {:registry cons-schema} ::cons])
this fails:
(dot/transform [:schema {:registry (merge (m/default-schemas) cons-schema)} ::cons])
; =>
:malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1}
I'm not sure how adding the defaults would affect things. : Related to this, what is the difference between these two?:
(m/schema [:schema {:registry cons-schema} ::cons]) ;; works
(m/schema ::cons {:registry cons-schema}) ;; fails -> :malli.core/invalid-schema {:schema :maybe}
#2021-05-1121:28borkdude@ben.sless @viebel Hope you guys are ok... don't know where you live, but it seems some bad stuff was happening in your country today#2021-05-1207:54Ben SlessThank you 🙂 Been an interesting night, but we're all fine. Hopefully it will not escalate further#2021-05-1204:10Yehonathan SharvitThank you for asking @borkdude. The situation is tough. Kids don’t go to school. We are a bit scared. Hopefully, we’ll be fine.#2021-05-1307:38Adam HelinsIs there something in the current design that could prevent implementing that kind of schema salvaging? This is about altering some properties of an existing schema in a direct way:
{:int-vec [:vector :int]
 :small-int-vec [:int-vec {:max 3}]}
I often have that kind of situation where I have a common schema (eg. :int-vec) but once in a while I have to put more constraints in properties or slightly alter generation (eg. {:max 3} ). Is there another obvious way, besides an external function that recreates the schemas with optional properties?
#2021-05-1307:55Adam Helins(and besides directly using malli.util)#2021-05-1311:50caumondHi everybody, I am looking for the documentation of :ref, I always have a doubt if it is needed or not. I find :ref in the example of the documentation but no sentence explaining why it necessary or not.#2021-05-1312:01ikitommi@adam678 interesting idea. Should the latter fail if it would have children too? Or swap those too, if present?#2021-05-1312:04ikitommi@caumond :ref is needed for recursion, no other utility I believe. It's implementation is lazy, so validators, explainers, generators etc. are realized only when needed.#2021-05-1312:04ikitommiDoc PR welcome#2021-05-1312:10caumond😁 I got it !#2021-05-1520:38caumondI posted a PR yesterday#2021-05-1312:07Adam Helins@ikitommi I was thinking of altering properties only since this is both useful and not controversial. Altering properties doesn't change fundamental aspects of a schema (usually, I guess). If we could alter children, the following would look weird, almost evil:
{:int-vec [:vector :int]
 :double-vec [:int-vec :double]}
#2021-05-1410:08Adam Helins(I opened an issue since it looks like it's worth having a discussion: https://github.com/metosin/malli/issues/448)#2021-05-1411:11Adam HelinsDid you consider a :multi variant where the function directly returns a schema as opposed to a mere dispatch value? Like Spec does, actually. Once in a while I have to write something like:
[:multi {:dispatch first} [::a ::a] [::b ::b] [::c ::c] ...]
#2021-05-1415:13snorremdHi. Is there a built in story for validating and coercing Record types? Using [:map [:fieldA :int] [:fieldB :int]] fails when Malli tries to humanise an error and calls the empty function in Clojure core on the record causing an error. Not sure if using the map schema is the correct way of going about this at all, or if it would be better to create a (malli.core/-simple-schema) or something. Any pointers would be appreciated.#2021-05-1710:16nilernI think :map matches the defrecord philosophy so the error you are getting would be a bug in error humanization. Humanization is not a fundamental operation so making records work should not be a big deal.#2021-05-1418:02Ivan FedorovI’ve gathered some observations about walking malli schemas to transform them into something (eql vector, clojure spec alpha). Critique and additions welcome https://github.com/dvingo/malli-code-gen/blob/main/src/dev/space/matterandvoid/transform_building_ideas.cljc If anyone experienced with malli has 30 minutes – let’s record a youtube video together on writing malli code transformations. I think, this should be valuable for those who want to write their own transformers and consequently the community.#2021-05-1713:26eoliphantHi, i’m not 100% clear on how far down the rabbit hole you can go in terms of defining schemas in terms of other schemas. I’m using a mutable custom registry ‘spec’ishly’ per the docs, and i’m running into some issues, but not sure why. Perhaps some ‘type’ v ‘instance’ thing? For something like the following:
(register! :bigdec (m/simple-schema ...))
(register! :money [:map {:someprop ..}
                     [::amount :bigdec] 
                     [::currency [:enum :USD :CAN ..]]])
(register!  ::annual-income [:money {:propa ..}])

(def x {::amount 1.23M ::currency :USD})
(m/validate :money x)
;=> true
(m/validate ::annual-income x)
;Exception => :malli.core/invalid-schema {:schema [:map {:someprop ..} ....
#2021-05-1716:53thomascothranQuestion about the schema walker functionality. I am trying to change all the :optional properties of a nested schema to false with this:
(mi/walk schema         ;; malli.core aliased as `mi`
    (mi/schema-walker
      #(mu/update-properties % assoc :optional false)))
However, this only works if maps have a single key. If the schema is something like this:
(def schema
    [:map
         [:teams
            [:vector
             [:map
              [:team
                 [:map
                  [:id uuid?]
                  [:name string?]
                  [:logoUrl {:optional true}
                   string?]]]
              [:health {:optional true}
                 [:map
                  [:injuries {:optional true}
                   [:map
                    [:playerInjuries
                     [:vector
                      [:map
                       [:playerName string?]
                       [:position {:optional true}
                        string?]
                       [:status string?]]]]]]]]]])
Only only the values under the :teams key are updated; the :health key and all the values under it are unaffected. Am I using the schema walker incorrectly, or is this possibly a bug? This is on version 0.5.1 Edit: After looking at this more, I think it's just that update-properties doesn't work on keys, so I was using it incorrectly.
#2021-05-1809:58Setzer22The syntax to define multi-arity fns is this one, right?
[:function
  [:=> [:cat int? int?] int?]
  [:=> [:cat int?] string?]]
#2021-05-1810:24Setzer22Ok, so I was asking because I'm implementing multi-arity schema support in my malli-instrument library, but I'm unsure how to best deal with instrumenting multi-arity fns with varargs. Due to the way in which varargs are specified, I don't have a way to know which of the arities under :function corresponds to each of the possible lengths of the arg list, or at least not without some analysis, consider the following:
[:function 
  [:=> [:* int?] int?]
  [:=> [:* string?] string?]]
this is a valid schema for:
(defn foo
  ([x y z] (+ x y z))
  ([a b c d & _] (str a b c d)))
it can be argued that it's an innacurate schema for the function, but that's not my point: Regardless of the example, the fact remains that with seqex patterns being used to define arglists, one can't easily determine which one of the annotated arities to take given an arglist and its length :thinking_face: The only way to do this with the current specification is to test each of the arity specs against the arglist until some validates. This is what I'm implementing now, but I thought it'd be good to bring out this discussion nontheless, in case you think the specification format could be improved 👍
#2021-05-1811:08ikitommi@setzer22 please check https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L264-L272#2021-05-1811:09ikitommiand; https://github.com/metosin/malli/blob/master/src/malli/clj_kondo.cljc#L134-L154#2021-05-1811:12Setzer22I'm confused, what's this second one doing? 😅#2021-05-1811:10Setzer22nice, thanks! 👍 I was re-inventing the wheel for some of this work in a more inefficient way so this helps a lot#2021-05-1813:49Setzer22just noticed a small inconsistency when writing the instrumentation code: (m/validate :cat nil) returns false, but when calling (fn [& args]) with zero arguments, args is bound to nil#2021-05-1813:51Setzer22I can easily work around it on my side using something like (or args '()), but I thought that may be something worth looking into#2021-05-1813:54Setzer22alright! malli-instrument now supports multi-arity fn schemas 😄 🎉 https://github.com/setzer22/malli-instrument I'll continue working on this and pushing bugfixes#2021-05-2009:05Michaël SalihiHi! Is Malli a good candidate to use for server side form validation in replacement for lib like https://funcool.github.io/struct/latest/? Especially thanks to malli.error / humanize?#2021-05-2009:10Michaël SalihiWith some hand made regex validators (email, etc) like https://github.com/funcool/struct/blob/master/src/struct/core.cljc#L195 + locale support, I think so, right?#2021-05-2009:50ikitommi@admin055 definetely#2021-05-2009:54Michaël SalihiPerfect, thanks for the confirmation. I'll toy with Malli in a project and see how to organize this. 👍#2021-05-2009:55Michaël Salihiis there an example demo project with form validation somewhere?#2021-05-2009:59Michaël SalihiI like very much the Reitit's example folder, if my tests are successful after a code review, I may be able to contribute with an example PR with form validation on Malli's Github.#2021-05-2009:59Michaël SalihiWhat do you think?#2021-05-2010:00ikitommisure, examples welcome! I have a malli-form (reagent) demo draft, but nothing serious.#2021-05-2010:02Michaël SalihiPerfect, let's do this!#2021-05-2010:07dharrigan has form validation#2021-05-2010:08dharrigan#2021-05-2010:08dharriganused here #2021-05-2010:10Michaël SalihiAwesome, thanks @dharrigan!#2021-05-2010:10dharriganyou're most welcome#2021-05-2016:18shanhi, Anyone know if Malli has support that allows you to set a default value if the value doesn't pass the schema check?#2021-05-2016:20dharriganYou mean this? #2021-05-2016:33shanlooks like that just sets a default if the value is missing:
(m/parse
 [:and {:default 42} int?]
 "test");; => :malli.core/invalid


(m/decode [:and {:default 42} int?]
          "test"
          mt/default-value-transformer);; => "test"
I'll probably just use m/explain and do something based on if there are errors
#2021-05-2104:49ikitommimight not be simple as any fix could change the conditional branch that should be used (with :or and :multi). But you should be able to use malli.util functions to access the invalid paths and work from there.#2021-05-2016:21shanpossibly. Generally I just want to check a map is valid but keep any good values and log out any errors#2021-05-2018:35Vladislavhi! Deref of [:merge ...] makes :registry with nested schemas of one of merged schemas invisible (but it still works). is it a bug, or feature, or there is some workaround Im missing? point is - registry disappears from any of serialisations after deref, although it contains all information about recursive fields, and i assume there is no other way of define recursive schemas. of course, i still could use underefed schema, but it much complicated#2021-05-2018:55ikitommiCan't recall where it was discussed, but we should clearly separate the locally registered schemas from normal registered ones and with the malli.util fns, expose the accumulated local registry, to be used in schema form. This makes the schemas visible and thus, serializable.#2021-05-2018:55ikitommismall change, clears thing a lot.#2021-05-2019:04VladislavI tried to find some way of extract and then serialize registry from schema (`m/-registry`, and then malli.registry/schemas ), but sadly unsuccess - it still needs more class serialization (may be to deref every schema from it? - leave it for tomorrow)#2021-05-2214:24Adam HelinsIf by any chance you ever had trouble with recursive data, please consider having a look at https://github.com/metosin/malli/issues/452 It is quite a hard problem and I would appreciate ideas, opinions, and reports if you had that kind of problems#2021-05-2218:09Adam Helins(and spec doesn't solve it either)#2021-05-2309:34Ben Slessis there a way of providing a transformer for key names? i.e. I expect a key :foo but after transformation the key will be :bar?#2021-05-2414:56ikitommi@ben.sless check out malli.transform/key-transformer#2021-05-2718:14Ben SlessI'm looking at JSON Schema now and trying to write it out with malli and recursive schemas (with the intention of writing a transformer from it to malli) I'm using this BNF as reference https://cswr.github.io/JsonSchema/spec/grammar/ I'm wondering what's the best schema to pick for modelling it A lot of it is defined in terms of key-value pairs I thought I'd try modelling it as a union for convenience I defined
(defn -ref-u [& args] (into [:union] (map (fn [k] [:ref k])) args))
Then
;; JSDoc := { ( id, )? ( defs, )? JSch }
   ;; id := "id": "uri"
   ;; defs := "definitions": { kSch (, kSch)*}
   ;; kSch := kword: { JSch }
Becomes
::JSDoc (-ref-u ::id ::defs ::JSch)
   ::id [:map [:id {:optional true} [:ref ::uri]]]
   ::defs [:map [:definitions {:optional true} [:map-of :keyword [:ref ::JSch]]]]
But when it comes to n>=1 map schemas which are one-of types, I'm stumped
;; JSch := ( res (, res)*)
i.e., the map should have at least one res where it's defined as
;; res := type | strRes | numRes | arrRes | objRes | multRes | refSch | title | description
   ::res (-ref-u ::type ::strRes ::numRes ::arrRes ::objRes ::multRes ::refSch ::title ::description)
Ideally, I'd want some schema between map-of and map I'm also not 100% clear on the difference between union and merge and if they're even the right choice. Should I perhaps customize a -collection-schema?
#2021-05-2915:02Ben Slessanyone?#2021-05-2719:11borkdudeIs this a standard? https://jsonlogic.com/#2021-05-2804:57ikitommihave looked at it, but haven’t heard of anyone using. not super popular.#2021-05-2804:57ikitommibut, I nice idea. not sure how many ways there are to describe such things.#2021-05-2915:02Ben Slessanyone?#2021-05-2807:50Pragyan TripathiIs there a way I can use malli to generate datomic schema?#2021-05-2820:53eoliphantnot ‘out of the box’ per se but it’s pretty straight forward. depends on the way you define your schemas. We use the dynamic registry sort of like spec, so our attributes are first class which sort naturally maps to datomic, and then just map over the registry for our attrs, :{db/ident <attr> :db/valueType (malli-to-datomic …) …} theres of course specific stuff you’ll probably do for enums :my/enum [:enum :a :b] becomes norms for a :my/enum ref and :my.enum/a, etc. :map’s to refs, etc. We have our own custom options for additional customization when we generate.#2021-05-2820:56eoliphantalso, this video talks about this in general in terms of using malli to drive a lot of the rest of your app. https://www.youtube.com/watch?v=ww9yR_rbgQs&amp;t=696s#2021-05-2821:05eoliphantis there a way to muck with option values based on a predicate? in pseudo-malli…
[:map 
  [:a :boolean]
  [:b {:optional #(true? :a)} :boolean]]
#2021-05-2909:19Adam HelinsWhy is :ref not allowed in seqex? Are they truly incompatible or is it a goal implementing it later?#2021-05-2909:24ikitommiThere was a reason, @U4MB6UKDL should know thay#2021-06-0213:29nilernAllowing :ref would incidentally make context-free parsing possible, like in Spec. But that is much harder to specify and implement cleanly and efficiently e.g. I think left recursion drives Spec to a stack overflow. The implementation could be elegantly extended to do GLL parsing, like Instaparse. But supporting all CFG:s means supporting the ambiguous ones as well, so parse would have to return a seq of plausible parse trees etc. One could argue ambiguous schemas are undesirable anyway, but the best way to ban those is (some extension of) LR parsing and nobody wants to deal with shift-reduce conflicts either and many grammars are unambiguous but not LR (without manual mangling) 😩 I would say from a theoretical perspective there are some satisfactory options but it is not simply a matter of allowing :ref. From a UX standpoint it comes down to most programmers happily wielding regex but quickly getting confused with YACC (and even Instaparse when it comes to ambiguous grammars).#2021-06-0216:34Adam Helins@U4MB6UKDL That's a very thorough answer, thanks!#2021-05-2909:22ikitommi@eoliphant kinda like dependent schemas? I guess we could have a formal way of doing that, e.g. schema value -> schema mapping. There is already content-dependent schemas and some ways to do that at validation time#2021-05-2909:25ikitommiSee https://github.com/bsless/malli-keys-relations#2021-05-2909:26ikitommifor the simple key, :or or :multi would work too#2021-05-2909:26ikitommi1. ... or just :and with a :fn constraint#2021-05-2909:28ikitommibut with dependent schemas you could ask "what is the derived schema for this given value, against this (conditional) schema". Would be great.#2021-05-2909:28ikitommibut with dependent schemas you could ask "what is the derived schema for this given value, against this (conditional) schema". Would be great.#2021-05-2913:28eoliphantyeah i’d played around with :or and :multi, I guess :fn is the only route, was just trying to still leverage the ‘declared’ type#2021-05-3115:55respatializedApologies if I missed this while looking through the docs/source, but is there a way to make map schemas (and perhaps other schemas converted to the map syntax) self-documenting by adding something like:
(def my-map [:map [:k1 {:doc "A required key, typically used by the program to do foo."} :string] [:k2 {:optional true :doc "An optional key, typically used by the program for annotation purposes."} :string]])
#2021-05-3116:02respatializedIt looks like malli.util/to-map-syntax and malli.util/from-map-syntax work just fine on the example I just gave, but I suppose I'm still wondering if the :doc kw in the props map is intended for a different purpose or potentially reserved in some way.#2021-05-3116:43ikitommi@afoltzm using qualified keys is the safest, plain keys could be used for something special, but not going to add anything special for :doc. So, safe in practice. :description is picked up JSON Schema transformer, so that works too and is reserved for docs.#2021-06-0106:00docoutoAny idea when the 'schema' branch could be merged into master?#2021-06-0106:01docoutoI mean, are there major things preventing this from happening?#2021-06-0114:51Ben SlessI hate to use the M word, but I think the answer to https://github.com/metosin/malli/issues/304 is yes. If we can say malli is working with types, aren't container schemas monads, of sorts? maybe is Maybe, or is Either, etc#2021-06-0115:03Ben Slessmap and and would be sort of product types?#2021-06-0213:35nilernI think Functor is more relevant than Monad here. tuple is a product too, or and some other things are sums. But still it is often not possible to jump to conclusions based on category or type theory.#2021-06-0213:37nilernI would rather have something like mu/fmap than select-keys automagically working through maybe etc.#2021-06-0213:39nilern
(mu/fmap #(mu/select-keys % [:x]) [:maybe [:map [:x int?] [:y int?]]])
;=> [:maybe [:map [:x int?]]] 
#2021-06-0214:49Ben SlessI agree with the observation, but I think the answer is the other way around - select-keys should work via fmap instead of knowing the implementation details of the container it's operating on#2021-06-0214:52Ben SlessAnd the reason I thought of Monads and not just Functors is cases like [:maybe [:maybe T]] which could be joined, and even hairier situations where you have some combinations of containers#2021-06-0214:54Ben SlessAnd if schemas are some representation of types, all the schema manipulations in malli.util could be represented as type manipulations which are implemented by the schemas instead of being bespoke data manipulations#2021-06-0707:14nilernI especially hate it when you run through several layers automagically, like in Cats I think (fmap inc [(just 3) 2]) ;=> [(just 4) 3] or something and JS Promises are not monads because .then tries to be both bind and (flipped) fmap...#2021-06-0707:22Ben Sless> anything automagically I can't disagree with you there. These things should be explicit. And yet, don't you think malli could benefit from abstracting schema manipulation and composition to interfaces of types manipulation and composition instead of data manipulation? It lends itself to some weird phenomena, such as mu/get-in in a maybe-schema requires explicit reference to the path, so in [:maybe [:map [:a int?]]] int? is in [0 :a]. Is that the right way to go about it?
#2021-06-0709:14nilernI haven't thought much about the schema manipulation stuff. But maybe the non-semantic (`[0 :a]` etc.) approach is not that bad and in fact kind of idiomatic. And I think even dependently typed languages like Idris will just pattern match on the type syntax even though it is much more against their philosophy (at least if the kind of the type is Type). It is hard to say what feels right when there is so little prior art (that I am aware of).#2021-06-0709:25Ben SlessThere's stuff over in typed land which also seems like leaky implementation and not a correct design choice
T1 = A | B
T2 = C | T1
Is not equivalent to
T3 = A | B | C
because unions are tagged, although it's still just a union of sets of input fields. They should be equivalent
#2021-06-0711:16nilernIn a language with untagged unions those will be equivalent (or it is a bug). I don't really regard ADT:s as union types, especially since in most languages each variant can have multiple and even named fields. And you can also do something similar with structural variant types (e.g. OCaml) although as usual type inference complicates that (unification must work etc.).#2021-06-0711:26nilernBy the way one reason lens types are hard is that not all containers have the contained things as type parameters e.g. data Point = Point {x : Int, y : Int} clearly contains Int:s but cannot be made a Functor or otherwise extract that fact on the type level because it's just an opaque name like :point#2021-06-0712:30Ben SlessYou mean it can be behind a reference?#2021-06-0809:03nilernIn SML it would be like a reference but in Haskell/OCaml/Java it is just an abstract type. Even If there was a way to get to the Int part(s) it is different from functors where it is always the first argument and you also have to decide between x, y or both...#2021-06-1802:35Ben Slesshttps://clojurians.slack.com/archives/CLDK6MFMK/p1623959758175200#2021-06-0306:13ikitommi@pedroabelleira need to clean it up first, hopefully before summer vacations#2021-06-0306:14ikitommia proposed fix to look up errors from parent schemas, finally: https://github.com/metosin/malli/pull/462#2021-06-0306:14ikitommifixes #86:
(-> [:map
     [:foo {:error/message "entry-failure"} :int]]
    (m/explain {:foo "1"})
    (me/humanize {:resolve me/resolve-root-error-message-and-path}))
; => {:foo ["entry-failure"]}
#2021-06-0306:15ikitommithe new hook in humanize allows custom collector, here, it just traverses the parents to look for error definitons, which will override the more exact definitions.#2021-06-0306:17ikitommiwe could have a m/humanizer that would prepare the humanization and would be much faster, e.g.
(def Schema
  [:map
   [:foo {:error/message "foo!"} :int]
   [:bar {:error/message "bar!"} :int]])

(def humanize (me/humanizer Schema)

(-> Schema 
    (m/explain {:foo "1", :bar "1"}) 
    humanize)
; => {:foo ["foo!"], :bar ["bar!]}
#2021-06-0307:31Grigory ShepelevHello there. Need a little help. How do I malli/get-in with spec with registry? Suppose having the following structure
(def registry
  {"users"
   [:map
    [:telegram
     [:map
      [:id int?
       :is_bot boolean?]]]]
   "messages"
   [:map
    [:from {:optional true} [:ref "users"]]
    [:id int?]
    [:text {:optional true} string?]
    [:reply {:optional true} [:ref "messages"]]]})
And I've tried a lot of different combinations like:
(let [s (malli/schema [:schema {:registry registry} "messages"])
                        _ (print (malli/schema? s))] 
                    (u/get-in s [:id]))
true;; => nil
No success.
#2021-06-0308:27Ben SlessYou have three mistakes: • syntax error in registry definition (look at the telegram map) • the registry you pass to the schema constructor is incomplete, you need to merge it with the default registry • The path you get-in is wrong, schema-schemas are considered part of the path and their children are at key 0.#2021-06-0312:38ikitommishould malli default to allowing the default registry o be swapped? e.g. strict mode where it can’t (for those who want to be fully in control).#2021-06-0314:29Ben SlessI think it would be better to merge the provided registry with the default registry be default. I always forget that#2021-06-0316:32ikitommione option should be not to have defauts schemas, only way to get proper DCE for tiny lib size, e.g. busy frontends.#2021-06-0316:33ikitommicurrently the smallest usefull malli-bundle is 2.3kb (gzipped js)#2021-06-0316:33ikitommiwith default registry, it’t 37kb.#2021-06-0317:50Ben SlessHow is that measured? I never worked with cljs. Does the compiler eliminate dead code?#2021-06-0613:56ikitommithere is a guide how to do that (with shadow-cljs) in malli readme. In short: • unsed functions and protocol methods (and definitions) will be DCEd • multimethods and deffed values are not#2021-06-0613:57ikitommithis is the reason all malli registry parts are behind a function -> no-one calling it => get’s eliminated under advanced.#2021-06-0614:30Ben Slesshuh, cool#2021-06-0312:40ikitommiseems that all the malli-codebases I have worked with, have introduced a custom mutable registry…#2021-06-0312:40ikitommiyou can always do evil:
(reset! @#'mr/registry* (mr/mutable-registry registry*))
#2021-06-0313:20ingesolHi! I want to create a schema for a sequence like this
[{:type  :type1
  :attrs {:type1-prop 1}}
 {:type  :type2
  :attrs {:type2-prop 2}}]
:attrs schema will be different for different object types. There’s :multi, but the dispatch property :type is on the parent of the object inside :attrs. Is the problem description clear, and does anyone have a good/best practice for solving this with malli?
#2021-06-0315:56ikitommi@ingesol maybe:
(require '[malli.generator :as mg])

(mg/sample
  [:multi {:dispatch :type}
   [:type1 [:map
            [:type [:= :type1]]
            [:attrs [:map
                     [:type1-prop :int]]]]]
   [:type2 [:map
            [:type [:= :type2]]
            [:attrs [:map
                     [:type2-prop :int]]]]]])
;({:type :type1, :attrs {:type1-prop -1}}
; {:type :type2, :attrs {:type2-prop -1}}
; {:type :type1, :attrs {:type1-prop -1}}
; {:type :type1, :attrs {:type1-prop -1}}
; {:type :type1, :attrs {:type1-prop -2}}
; {:type :type1, :attrs {:type1-prop 0}}
; {:type :type2, :attrs {:type2-prop 12}}
; {:type :type2, :attrs {:type2-prop 4}}
; {:type :type2, :attrs {:type2-prop -26}}
; {:type :type1, :attrs {:type1-prop -14}})
#2021-06-0317:13ingesol@ikitommi Thanks! Yes, I thought about something like that too. My maps are bigger than in this minimal example, but easy enough to create the schemas with a factory function i guess.#2021-06-0319:36ChillPillzKillzBillzNewbie here... I am trying to run the base code example from the lambdaisland/regal example page https://github.com/lambdaisland/regal. My Deps.edn looks like {:deps {org.clojure/clojure {:mvn/version "1.10.3"}         `org.clojure/core.async {:mvn/version "1.3.618"}`         `org.clojure/test.check {:mvn/version "1.1.0"}`         `lambdaisland/regal {:mvn/version "0.0.97"}`         `metosin/reitit-malli {:mvn/version "0.4.2"}}` and my clojure code is as follows (ns baseClj.core   `(:require [malli.core :as m]`             `[malli.error :as me]`             `[malli.generator :as mg]`             `[lambdaisland.regal.malli :as regal-malli]`             `;; [lambdaisland.regal :as regal]`             `;; [lambdaisland.regal.generator :as regal-gen]`             `))` and first few lines (def malli-opts {:registry {:regal regal-malli/regal-schema}}) (def form [:+ "y"]) (def schema (m/schema [:regal form] malli-opts)) (m/form schema) my namespace definition gives me Error: "No namespace: lambdaisland.regal.malli"... but works when I remove the :as m . Then it fails on the lambdaisland.regal.malli, and again works when :as regal-malli is removed... Finally if fails at (def malli-opts {:registry {:regal lambdaisland.regal.malli/regal-schema}}) with the error ; Syntax error (ClassNotFoundException) compiling at (h:\Work\Clojure\baseClj\src\baseClj\core.clj:55:1). ; lambdaisland.regal.malli . None of this makes any sense... this is just the example code. What am I missing?#2021-06-0320:01ChillPillzKillzBillzI got a hint... The deps.edn include for malli was incorrect. I should have added metosin/malli {:mvn/version "0.5.1"} . This doesn't solve the problem tho...#2021-06-0323:05ribelohttps://github.com/metosin/malli/pull/305 is it dead or waiting for better times?#2021-06-0415:27ingesolI’ve been trying to get this to work, but cannot figure out what is going on. I’m expecting the :type value to be converted to a string:
(m/encode
 [:schema {:registry
           {::node [:multi {:dispatch      :type
                            :decode/string #(update % :type keyword)}
                    [:malli.core/default
                     [:map
                      [:type :keyword]]]]}}
  ::node]
 {:type  :doc}
 mt/string-transformer)

=> {:type :doc}
If I replace :malli.core/default with :doc, I get the expected string-coerced keyword
{:type "doc"}
Also, the schema has a local registry in order to be able to have nodes within nodes. If I remove the wrapping, the following also works:
(m/encode
 [:map
  [:type :keyword]]
 {:type  :doc}
 mt/string-transformer)

=> {:type "doc"}
#2021-06-0417:23datranWhat's the difference between parsing data and decoding data? My sense is that parse is similar to spec's conform, whereas decode is more about transformation - is that generally right?#2021-06-0417:39ikitommidecoding is a process of transforming values from external formats into valid clojure data. Parsing returns the parse trees.#2021-06-0417:44ikitommi@ingesol what version are you using. CHANGELOG says it's fixed in 0.5.0#2021-06-0417:44ikitommihttps://github.com/metosin/malli/issues/415#2021-06-0419:17ingesol@U055NJ5CC hmm, I thought I was on 0.5.2 or something, will check#2021-06-0419:23ingesolVery happy to be wrong, I was on 0.4.0. Thanks 🙂#2021-06-0419:29ingesolAaaand it works! Thanks for great help, as always#2021-06-1206:07robert-stuttafordis there a trick to debugging this m/explain exception? Vector's key for assoc must be a number. at https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L139#2021-06-1207:20ikitommihumaize should not throw, ever. There is an open issue/PR, seems stalled: https://github.com/metosin/malli/pull/333#discussion_r588855299#2021-06-1306:22robert-stuttafordi'll put together a repro and submit an issue. my problem seems different to the one that you have linked#2021-06-1314:46robert-stuttafordturns out i was overwriting instead of adding to the default error message set facepalm#2021-06-1215:08respatializedI feel like I may have asked this question before, but am having trouble finding the right keywords to search in the slack archive, so apologies for the redundant question: Does malli provide a meta-schema or helper function I can use to programatically validate forms that may or may not be valid malli schemas? Is there a self-describing malli schema for malli schemas? Something like: (m/validate malli-meta-schema [:a]) => false (m/validate malli-meta-schema [:or [:fn int?] [:fn string?]]) => true#2021-06-1215:42ikitommi@afoltzm not yet. But since 0.5.0, each IntoSchema can describe it's properties and children schemas. Once all built-in schemas have described themselves (currently empty impls), one can ask from a registry the accumulated schema for schema AST.#2021-06-1215:47ikitommisee m/children-schema and m/properties-schema#2021-06-1216:06respatialized@ikitommi gotcha, thanks! right now I'm just doing a basic helper predicate that relies on the catching the exception thrown by calling m/schema on the form, which I think suffices for my current use case.#2021-06-1218:14borkdudeEnjoyed this podcast: https://twitter.com/JacekSchae/status/1402648727215542272#2021-06-1315:09robert-stuttafordis there a way to do this :enum {:error/fn entity.validation/malli-humanize-enum-error} for all enums in the spec, perhaps when calling m/humanize ? otherwise i could walk the spec and inject them#2021-06-1317:12ikitommi@robert-stuttaford humanize takes an :errors option, where you can override :enum handling. Walking is another way to do it. There might be others too#2021-06-1318:25robert-stuttafordthanks @ikitommi - can you point me to example of the :errors option specifically overriding the printing of errors for a specific spec type like :enum that i can work from, please?#2021-06-1319:08ikitommisure @robert-stuttaford, https://github.com/metosin/malli/blob/master/test/malli/error_test.cljc#L177-L188#2021-06-1320:10robert-stuttafordah so:
(assoc-in me/default-errors [:enum :error/fn :en] my-fn-here)
#2021-06-1320:10robert-stuttafordthanks!#2021-06-1411:18ikitommiI would say (assoc me/default-errors :enum {:error/fn {:en my-fn-here}}) just to make sure that the whole :enum map is overridden, can’t recall which one is picked first, :error/fn or :error/message if both are present#2021-06-1421:43martinklepschHey 🙂 Is there a way to combine multiple schemas to one? Specifically I have a bunch of “base” keys and a few keys that I want to check based on the :type key in the set of “base” keys. I kind of see how I could do that with multi and a bit of repetition but I probably want :merge? Is there a way to get :merge without getting into registries and all that?#2021-06-1421:50martinklepschDuh, I just realized I can just into more key to existing [:map] schemas#2021-06-1504:45ikitommiyeah. :multi is not currntly mergable, oftenly asked. would allow concise definitions like:
[:merge
 [:map [:id :uuid]]
 [:multi {:dispatch :type}
  [:pear [:size :int]]
  [:boat [:price :int]]]]
#2021-06-1615:03martinklepschIs there a “recommended way” to achieve this currently?#2021-06-1615:03martinklepschIt seems like a common need so if there is one I think documenting that could be useful. Happy to open a PR once I know 😄#2021-06-1507:07wcalderipeHey folks 👋 Why explain keeps passing values forward when previous operations in an and condition have already failed?
(m/explain
   [:map
    [:name
     [:and string?
      [:not {:error/message "non empty string"} empty?]]]]
   {:name 1})
The example above throws Don't know how to create ISeq from: java.lang.Long. It does at (empty? 1). The conditions are concatenated with an and operator, so I wasn't expecting [:not {:error/message "non empty string"} empty?] to run because string? failed in the first place. Note: there are probably better ways to express a non-empty string with Malli 😅
#2021-06-1507:49Martín Varela[:string {:min 1}] is probably what you're after...#2021-06-1508:29ikitommiyes, I think it’s better to have keyword types (e.g. :string) with properties than to use the existing core functions, of which, many throw on invalid input.#2021-06-1508:29ikitommie.g. post-int? vs [:int {:min 1}]#2021-06-1511:46wcalderipe@U95NTJT4H thanks for the suggestion – I knew there was a better way to solve that problem.#2021-06-1511:47Martín VarelaGlad that helped 🙂#2021-06-1511:47wcalderipe@U055NJ5CC got it. For custom keyword types, we would have to provide a registry, right?#2021-06-1511:51wcalderipeSo, I assume Malli will check the value against every keyword type of a field, even when using an AND operator and the first check fails, in order to get a complete errors map in the first run :thinking_face:#2021-06-1514:31ikitommiyes, for the last one. See https://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&amp;schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D.#2021-06-1517:04wcalderipeGot it.. thanks @U055NJ5CC#2021-06-1519:20VladislavHi! I try to use malli as coercion&schemas with swagger ui. If i try serialise recursive schemas with registries, i get some "Could not resolve reference" of keys of recursive values. it is according to plan and swagger bad, or its just undone feature?) looks like there is some missing defs in swagger.json#2021-06-1519:43ikitommi@UNJE65T24 It might be that the references should copied to top-level swagger. If you can provide a minimal repro, would help solving#2021-06-1519:43ikitommialso, what version of swagger-ui are you using?#2021-06-1520:47Vladislav@U055NJ5CC i use swagger-ui bundled with
[metosin/reitit-swagger-ui "0.5.13"]
[metosin/ring-swagger-ui "3.36.0"]
guess it 3.XX
#2021-06-1520:49Vladislavi'v got example of that malli#2021-06-1520:54Vladislavif i malli.json-schema/transform this (it is used in swagger gen, right?) i get something like#2021-06-1520:54Vladislavand it has no valuable
#/definitions/Expression
as you can see
#2021-06-1607:00Vladislavshould i submit an issue to github?)#2021-06-1607:05ikitommiplease do#2021-06-1607:11Vladislavvoila https://github.com/metosin/malli/issues/464#2021-06-1608:03Yehonathan SharvitWhat's the rationale behind the fact that a set is considered as :sequential ? For instance
(validate [:sequential string?] #{"aa"}) ;; false
#2021-06-1608:19ikitommiit’s following the clojure way:
{:sequential (m/-collection-schema {:type :sequential, :pred sequential?})}

(sequential? #{}) ; => false
#2021-06-1608:20ikitommiyou should be able create custom collections types easily that access both sequentials and sets.#2021-06-1608:50Yehonathan Sharvit@ikitommi Could you share a pointer to the documentation about custom collection types?#2021-06-1609:16ikitommisure: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L813. Documentation PRs most welcome 🙂#2021-06-1609:16ikitommiin use: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1925-L1927#2021-06-1610:06Yehonathan SharvitHow it would look like to create a custom :sequential-or-set predicate? And where would i put its definition?#2021-06-1615:45ikitommi1. def a Var and use it instead of the type (keword), like with Reagent 2. add it to a registry, see README for alternatives#2021-06-1615:46ikitommi
(def SequentialOrSet (m/-col...))

(m/validate [SequentialOrSet :int] [1])
#2021-06-1706:04Yehonathan SharvitOk. Thanls#2021-06-1611:23borkdude@viebel sequential means: it has a defined order. sets are not ordered, similar to maps, although you can create sequences out of them#2021-06-1614:08Yehonathan SharvitYeah. It makes sense. But still surprising.#2021-06-1614:08Yehonathan SharvitI think we need a name and a predicate for a bunch of things that could be either in a set or in a sequence#2021-06-1614:09Yehonathan SharvitMy use case is a function that receives a bunch of ids and return the entities whose id is contained in the bunch#2021-06-1617:53ikitommiyou could always say [:or [:sequential :int] [:set :int]].#2021-06-1617:53ikitommior, one could always transform the values always to sets/vectors.#2021-06-1618:05eskoscoll? works as well, if you don't mind using predicates directly#2021-06-1618:30ikitommi(coll? {}) ;=> true#2021-06-1618:33ikitommicoercion is always an option:
(m/decode
 [:set :int]
 [1 2 3]
 (mt/collection-transformer))
; => #{1 2 3}
#2021-06-1621:10deadghostAny recommendations on how to approach extracting out a registry from a schema? For example:
[:map
  [::id int]
  [:name string?]
  [::country {:optional true} string?]]

to

{::id int?
 ::country string?}

and as a nice to have, the schema simplified to use the new registry:

[:map
  ::id
  [:name string?]
  [::country {:optional true}]]
Use cases: • Simplify large schemas • Finding differences in semantics • Refactoring multiple schemas to use a shared registry
#2021-06-1702:07escherizeHello, I want to build a general data-building ui, that takes a description of the datastructure as input (a malli schema, this time around). I couldn’t find much prior work in this regard. I have a half-baked prototype here: https://escherize.com/w/data-desk/ So far, that mostly works for int? string? boolean? :vector and :map and combinations of them. So do you know of anything similar?#2021-06-1702:43escherizeAs for the implementation, I’m using a local atom and this multimethod: https://github.com/escherize/data-desk/blob/main/src/data_desk/views.cljs#L37#2021-06-1706:05Yehonathan SharvitI'd like to validate inside a function that the function is called with valid arguments. Should I use :=> ? How exactly?#2021-06-1706:21ikitommiREADME should cover that#2021-06-1706:21ikitommihttps://github.com/metosin/malli#function-schemas#2021-06-1709:12Yehonathan SharvitThe readme doens't cover it#2021-06-1712:48Yehonathan SharvitHere is what I am doing#2021-06-1712:51Yehonathan Sharvit
(def =>plus [:=> [:cat int? int?] int?])

(defn plus [x y]
  (when-not
      (m/validate (second =>plus) [x y])
    (throw (ex-info "invalid input" {})))
  (+ x y))
#2021-06-1712:51Yehonathan SharvitIs there a more idiomatic way?#2021-06-1715:58ikitommithere will be malli.instrument, while waiting, there is malli-indtrument - https://github.com/setzer22/malli-instrument#2021-06-2007:12Yehonathan SharvitVery cool! Any reason why it's not yet part of malli?#2021-06-2007:16ikitommijust time to merge & cleanup, the original issue is here: https://github.com/metosin/malli/issues/349#2021-06-2007:18ikitommithe function checker code shuould be reused between malli.generate and malli.instrument.#2021-06-2107:43Yehonathan SharvitIs someone already working on this consolidation?#2021-06-2108:10ikitommiplease ask on the issue. on my summer backlog if no-one has time. need this too after the vacations.#2021-06-2209:07Yehonathan Sharvitok. will do#2021-06-1719:55deadghostIs there something I can do to make the results of mu/union more condensed?
(mu/union (mu/union nil? empty?) empty?)
=> [:or [:or nil? empty?] empty?]
#2021-06-1804:53ikitommi@escherize looks good! haven’t seen anything library-quality impls of malli-forms. Have done one, but not complete and coupled to the use case.#2021-06-1804:55ikitommi@deadghost there should be an optional optimizer in malli. @miikka did a prototype of such some time ago: https://github.com/miikka/boolean-simplifier#2021-06-1804:57ikitommi[:or [:int {:min 10}] [:int {:max 5}] [:int {:min 0, :max 20}]] could be just :int.#2021-06-2006:49ikitommihttps://github.com/metosin/malli/blob/master/docs/tips.md#collecting-inlined-reference-definitions-from-schemas, ping @deadghost#2021-06-2117:52ikitommimalli.plantuml , the 23 loc monster 🙂#2021-06-2117:53ikitommihttps://github.com/metosin/malli/pull/468#2021-06-2118:00emccue> #(apply println %&)#2021-06-2118:00emccuewhat is that syntax#2021-06-2118:01emccue%&#2021-06-2118:01dvingohttps://clojure.org/guides/weird_characters#_n_anonymous_function_arguments#2021-06-2216:36ikitommi#(apply println %&) = println awesome#2021-06-2119:29alpoxOut of interest: Is there something in malli like a metaschema that can be used to validate a malli schema?#2021-06-2119:47Ben SlessMetacircular malli?#2021-06-2120:02alpoxMetacircular? 😄 You got me confused#2021-06-2120:18alpoxI just read up on Metacircular. I guess I am talking about something in that direction 😉 Basically a Malli schema that can be used to validate if my data at hand is a valid malli schema. If it was just a special keyword to refer to the metaschema while the metaschema is not directly written as a malli schema itself that would probably also be usable enough.#2021-06-2120:58ikitommiping @alpox#2021-06-2121:03alpox@ikitommi Oh, I really should have seen that in the history.. Thanks, that sounds promising!#2021-06-2209:08Yehonathan SharvitWhat's the recommended location for function schema definition? Should it be on the same namespace as the function or in a separate namespace?#2021-06-2214:27ikitommiNo strong opinions in malli for that. My favourites: 1. Inlined (the plumatic syntax) 2. just before the functions (`m/=>` is as long as defn , looks good#2021-06-2214:27ikitommiwhat do you think?#2021-06-2214:45Yehonathan SharvitSomeone reviewed my code that added m/=> for each and every function in the core namespace of a lib (around 10 functions) and he thought it was polluting the namespace#2021-06-2214:46Yehonathan SharvitI tend to prefer to have the schemas near the code as schemas serve also as documentation#2021-06-2215:13eskosI prefer the Plumatic style as it's more about upfront (meta) data expression, m/=> feels unnecessarily clever and sort of indirect/IoC hellish since it comes after the function declaration. But that's just my two cents 🙂#2021-06-2314:38Lucy Wang+1 for this. Being able to place the schema right near to the argument is way more intuitive and maintainable than having them separate from each other.#2021-06-2215:14emccueOne option would be this#2021-06-2215:14emccuehttps://github.com/galdre/morphe#2021-06-2215:15emccuemake an aspect for it#2021-06-2215:16emccue
^{::m/aspects [(contract [:=> [:cat int? int?] string?])]}
(m/defn some-fn [x y]
  (str (+ x y))
#2021-06-2215:22Yehonathan SharvitDoes malli support plumatic style? how?#2021-06-2216:26ikitommiwip, https://github.com/metosin/malli/pull/305#2021-06-2317:16respatializedI'm wondering whether function metadata maps may be suited to this purpose, the way spec does with :pre/:post validation inlined in an ordinary defn form, without needing a macro for function definitions. There may be a good reason to not overload the "canonical" keywords and instead use something like :malli/schema to inline it, but having it right in the form using metadata allows tests to be easily derived from a self-describing function with meta. https://clojure.org/guides/spec Was there something in your experience with spec-tools that led you to believe this isn't a good idea or the optimal solution for malli?#2021-06-2317:51ikitommithe plumatic syntax has been out since 2013 and I believe it's the most used syntax, by far. I have used it, it's just good.#2021-06-2317:53ikitommiI like also the`:malli/schema` suggestion, the malli.instrument could be configured to support multiple sources like: 1. the => registry 2. the :malli/schema metadata 3. plain old function schema inferrer#2021-06-2317:54ikitommiand, to be compliant: 4. the spec registry#2021-06-2216:29Yehonathan SharvitOk#2021-06-2216:31Yehonathan Sharvit@ikitommi In your previous message, you wrote: > just before the functions (`m/=>` is as long as `defn` , looks good What do you mean by m/=> before the function? In malli's README, m/=> comes after the function definition#2021-06-2216:35ikitommithis works too:
(m/=> plus [:=> [:cat [:* :int]] :int])
(defn plus [& ns] (apply + ns))

(plus 1 2 3)
; => 6
#2021-06-2216:37Yehonathan SharvitNice. In fact malli treats function names as symbols#2021-06-2216:38Yehonathan SharvitSo it doesn't matter if you create the schema before the function is defined#2021-06-2314:38Lucy Wang+1 for this. Being able to place the schema right near to the argument is way more intuitive and maintainable than having them separate from each other.#2021-06-2317:16respatializedI'm wondering whether function metadata maps may be suited to this purpose, the way spec does with :pre/:post validation inlined in an ordinary defn form, without needing a macro for function definitions. There may be a good reason to not overload the "canonical" keywords and instead use something like :malli/schema to inline it, but having it right in the form using metadata allows tests to be easily derived from a self-describing function with meta. https://clojure.org/guides/spec Was there something in your experience with spec-tools that led you to believe this isn't a good idea or the optimal solution for malli?#2021-06-2321:31escherizeI’d like to attach error messages to specific parts of my schema. So far I have the following working, but I’m not sure if it’s the best implementation.
(def MySchema
  [:map
   [:something/one {:error/message "one"} int?]
   [:something/two {:error/message "2"} int?]
   [:something/three {:error/message "three"} int?]
   [:something/four {:error/message "four"} int?]])

(validate MySchema {})
;; => #:something{:one ["one"], :two ["2"], :three ["three"], :four ["four"]}
#2021-06-2321:33escherizeI am using the following to get this working:
(defn find-error-message [schema k]
  (let [value (atom nil)]
    (walk/postwalk
     (fn [x]
       (when (and x (vector? x) (= (first x) k) (map? (second x)))
         (reset! value (second x)))
       x)
     schema)
    (:error/message @value)))

;; then, in humanize:

(me/humanize ...
 {:errors
  (assoc me/default-errors

   :malli.core/missing-key
   {:error/fn (fn [{:keys [in]} _]
                (or
                 (find-error-message schema (last in)) ;; is (last in) reliable here?
                 (str "Missing key: '"(last in)"'")))})})
#2021-06-2321:38escherizeI’m not sure if using (last in) in the :error/fn is reliable though.#2021-06-2404:56ikitommi@escherize maybe:
(-> [:map
     [:something/one {:error/message "one"} int?]
     [:something/two {:error/message "2"} int?]
     [:something/three {:error/message "three"} int?]
     [:something/four {:error/message "four"} int?]]
    (m/explain {})
    (me/humanize {:resolve me/resolve-root-error}))
; => #:something{:one ["one"]:two ["2"], :three ["three"], :four ["four"]}
#2021-06-2404:58ikitommithat’s in master, but not released. can be used, but will most likely change it to be a separate step, something like:
(-> Schema
    (m/explain {})
    (me/root-causes)
    (me/humanize))
; => #:something{:one ["one"]:two ["2"], :three ["three"], :four ["four"]}
#2021-06-2405:01ikitommianyway, it’s generic, so will find the top-most error defined:
(-> [:cat {:error/message "oh no!"}
     [:? [:map-of :string :any]]
     [:* :boolean]]
    (m/explain 123)
    (me/humanize {:resolve me/resolve-root-error}))
; => ["oh no!"]
#2021-06-2405:02ikitommithere is a perf hit of recurring all errors towards root, but it can be later removed with an extra step of preparing a me/humanizer.#2021-06-2405:57escherizeAwesome. I’ll look into this. Thanks @ikitommi #2021-06-2501:55pinealanHi there! I’m trying to use malli as a parsing+coercion library for external data. I’m a bit lost on what the schemas should look like and what function should I be using. An example is I want to transform a map with shortened key names like this {"a" "11", "b" "10.01"} into this {:ask 11.0 :bid 10.01}. Another case is sequential data, which may look like this ["11" "10.01"], and I would want to get the same data back. I know it’s possible to do some of these steps without malli, i.e. rename-keys or spec/explain with :sequential, but it felt like there should be builtin idioms from the library/schema definitions that helps with the extra transformations that I am performing. Am I just missing something?#2021-06-2507:03ingesolI’m no expert, but I think you want encode/decode. See https://github.com/metosin/malli#value-transformation. You could write your own custom key-transformer that maintains a mapping of the abbreviations you want.#2021-06-2507:04ingesol
(mt/key-transformer {:encode your-ns/kw->abbr
                     :decode your-ns/abbr->kw})
#2021-06-2507:07ingesolSchema for map would be
[:map-of :keyword :number]
Or whatever more specific number type you want.
#2021-06-2507:07ingesol^^^ @achan961117#2021-06-2507:11ingesolSomething like this in the end (non-tested code)
(m/encode [:map-of :keyword :number]
          {:ask 11.0 :bid 10.01}
          (mt/transformer
           (mt/string-transformer)
           (mt/key-transformer {:encode your-ns/kw->abbr
                                :decode your-ns/abbr->kw})))
;; =>> {"a" "11", "b" "10.01"}
#2021-06-2513:47Gary BergerHello when using map syntax is there a way of describing a oneOf relationship such that:
[:map
  [:or
    [:opt1 integer?]
    [:opt2 integer?]]]  
will allow either :opt1 or :opt2?
#2021-06-2513:55emccuedo you intend that or to be inclusive or exclusive#2021-06-2513:57emccue
[:and
 [:or [:map [:a int?]]
      [:map [:b int?]]]
 [:map [:x int?] [:y int?]]]
#2021-06-2517:35Gary BergerJust for clarity and not sure there is an easier way but exclusive or looks like
(m/validate
  [:and
   [:or
    [:map [:p integer?]]
    [:map [:q integer?]]]
   [:not
    [:and
     [:map [:p integer?]]
     [:map [:q integer?]]]]]
  {:p 123 :q 123}) => false
#2021-06-2513:58emccuei'm curious if there is a better answer that generates gooder, but this would do the right validation#2021-06-2516:15Gary Bergerexclusive thanks @emccue#2021-06-2517:09ikitommi@garyberger Usually, :and and :or are the way to go, see example in http://malli.io but there is also a declarative map-keys-relations poc: https://github.com/bsless/malli-keys-relations#2021-06-2821:44borkdudeIn case you didn't see it yet: https://github.com/piotr-yuxuan/malli-cli#2021-07-0115:36ikitommiThanks! Added link to README#2021-06-2822:47Gary BergerI will open an issue but this spec returns NPE with explain
(deftest xor-test
  (is (= false
         (specs/explain
          [:and
           [:or
            [:map [:network integer?]]
            [:map [:snapshot integer?]]]
           [:not
            [:and
             [:map [:network integer?]]
             [:map [:snapshot integer?]]]]] {}))))
#2021-07-0115:36ikitommiAnswered, could not reproduce with malli 0.5.1#2021-07-0115:36ikitommiThanks! Added link to README#2021-06-3009:31Richard HundtHi everyone. I've got a malli schema of around 1500 sloc, with quite a lot of :multi schemas. I'm using it for transformation, and to speed up the runtime, I'm using malli/encoder and malli/decoder , however creating these transformers takes over a minute with version 0.5.1. I'm not sure how to reason about the complexity there, as it seems like I've got some combinatorial explosion (start-up time increases non-linearly in the size of the schema). Has anyone else run into this sort of thing? Any tips? EDIT: I don't really see the need for speculative parsing or backtracking, since all the :multi schemas have discriminators which :dispatch can use to disambiguate, but there must be a lot of that going on, because plain calls to malli/decode without the "compiled" versions returned by malli/decoder take minutes themselves (not using regex schemas at all, either). 😕 (I'm running an Intel(R) Core(TM) i7-10875H)#2021-07-0115:39ikitommithat's not good @U026F3GM8NA. Could you create an issue out of this with a repro code showing the slowness. Thanks#2021-07-0118:03Richard HundtWill do.#2021-07-0206:29Ben Sless@U055NJ5CC could it be related to into transformer using satisfies? It's notoriously slow and GC intensive#2021-07-0207:16ikitommiit could, easy to see from flamegraphs.#2021-07-0207:40Ben SlessI can already guess from my experience with recursive calls to satisfies? , you'll see 95% of CPU is wasted on it#2021-07-0211:35Ben SlessIf -into-transformer was turned into a protocol, how would you make it work with cljs? fn? there is very different from in clj and won't be covered by a simple extend. Possible to do in cljs what was done for Schemas in clj?#2021-07-0113:01pithylessIs there an existing PR or issue related to "dependent schemas" (i.e. returning a derivative schema, based on schema and value) that was mentioned on Jacek's podcast?#2021-07-0115:49ikitommiI don't think there is an issue yet.#2021-07-0115:51Ben SlessDo you feel like there might be a need for a getting started guide? While the readme is comprehensive, I get the impression new users have a hard time building up beyond the simple examples. Something which starts from a simple base case, expands on it and illustrates malli's capabilities and features might be useful#2021-07-0116:07ikitommiThat would be awesome. Everything under docs gets published to cljdoc. Could also be a separate documentation site, if someone has time to work with that.#2021-07-0116:08ikitommiexamples of good doc sites (and tools to create them) would be a good start#2021-07-0116:28respatializedstay tuned on that one, I'm working on a heavily malli driven static website generator that uses itself to document itself#2021-07-0116:15Ben SlessDocs and cljdoc are good enough imo, but I'm more of a content over presentation kind of guy#2021-07-0116:18Ben SlessI'll start writing something. do you prefer adoc or md format?#2021-07-0116:18Ben Slessand on another note, any updates regarding the performance MRs I opened?#2021-07-0116:30ikitommiboth formats good, whatever you prefer. Adoc is better, right? The perf PRS (and all others), will check those soon. Just started my vacation, should have more time to invest in malli now.#2021-07-0116:55Ben SlessAren't you supposed to take time off on vacation? 🙂#2021-07-0116:56Ben Slesswho am I kidding if I had time off I don't think I'd be able to keep away from the computer for more than a day#2021-07-0117:14ikitommi6 weeks, should have time for both 😉#2021-07-0117:24Ben SlessLooks wonderful. Have fun!#2021-07-0117:51borkdude6 weeks, wow#2021-07-0123:12gregIs it possible to write mali, convert it somehow to spec and use it for spec/fdef and orchestration?#2021-07-0209:19ikitommino atm, but could be. Thing is that you can present things with malli that are impossible to present as specs, so it would not be complete like malli->JSON Schema , which is lossy. Still, I think 95% would work ok. There is malli-instrument library for doing the same with malli, if that’s ok for your use case.#2021-07-0214:09gregMakes sense. I wouldn't mind losing some of the malli features when instrumented as a spec. Some of the libraries (like integrant's pre-init-spec) support only spec. Although in my case, I use spec mostly for instrumentation of some bits. I was asking for such a conversion, as it sounds like the non-invasive way of trying malli in an existing project. Anyway, I will take a look at malli-instrument. Thanks.#2021-07-0207:18rmcvHow do I provide sci options in malli decode? Trying to use java class with no luck.
(m/decode [:int {:decode/string '(fn [_] (.toEpochMilli (java.time.Instant/now)))}]

          "whatever"

          {:classes {'java.time.Instant java.time.Instant}}

          mt/string-transformer)
#2021-07-0209:21ikitommilooking at the code, it’s :malli.core/sci-options key in options, see https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L1792-L1813#2021-07-0214:53rmcvtried this
(m/decode [:int {:decode/string '(fn [_] (.toEpochMilli (java.time.Instant/now)))}]
          "whatever"
          (merge
           (m/-default-sci-options)
           {:malli.core/sci-options {:classes {'java.time.Instant java.time.Instant}}})
          mt/string-transformer)
#2021-07-0214:53rmcvbut can’t see it in https://github.com/metosin/malli/blob/eea10aa8c668a16f88d9fc66fce0616a7f9137b4/src/malli/sci.cljc#L10#2021-07-0209:41ikitommi@ben.sless merged the loop unrolling perf PR, great for common cases and as it’s just clj, doesn’t make the cljs bundle size any bigger, big thanks!
;; 164ns -> 36ns
(let [valid? (m/validator [:and [:> 0] [:> 1] [:> 2] [:> 3] [:> 4]])]
  (cc/quick-bench (valid? 5)))
#2021-07-0210:45ikitommimerged the other perf PR too:
;; 150ns -> 39ns
(let [valid? (m/validator [:map [:a :any] [:b :any] [:c :any] [:d :any] [:e :any]])
      value {:a 1, :b 2, :c 3, :d 4, :e 5}]
  (cc/with-progress-reporting
    (cc/quick-bench (valid? value))))
#2021-07-0210:54ikitommi#2021-07-0211:22Ben SlessNice 😁#2021-07-0509:47Richard Hundt@U055NJ5CC I've created https://github.com/metosin/malli/issues/472 with a test case which demonstrates the issue.#2021-07-0511:01Ben Sless@U026F3GM8NA I profiled you example, it looks like CPU mostly goes towards parsing the schema. If you wrap every def-ed schema with a schema constructor it seems to be way more efficient, would you mind trying?#2021-07-0511:02Richard HundtD'you mean just wrapping it in a function which returns the structure? (I'm not sure what you mean by "schema constructor")#2021-07-0511:06Ben Slesslike so;
(def A1
  (m/schema
   [:map
    {:encode/test {:leave encode} :decode/test decode}
    [:type [:= "A1"]]
    [:value float?]
    ]))
#2021-07-0511:06Richard Hundtthanks, I'll try#2021-07-0511:06Ben Slessm/schema parses the children once#2021-07-0511:06Ben Slessyour implementation has no way to cache these results#2021-07-0511:07Ben SlessIt's a bother to do from scratch, you can take this#2021-07-0511:07Ben Sless#2021-07-0511:08Ben Slessthanks slack, very helpful#2021-07-0511:08Richard Hundtthank you#2021-07-0511:08Ben Slesshttps://gist.github.com/bsless/28a20f7e8f21f01fd93a788cf65444d4#2021-07-0511:09Ben Slessthere, more convenient#2021-07-0511:09Richard Hundtyeah, that's pretty much instant now, I'll try it on our production code#2021-07-0511:19Ben Sless👍#2021-07-0511:36Richard HundtPerfect. Thanks Ben! What shall I do with the github issue? Do I close it, or are you planning on making changes to parsing?#2021-07-0512:43ikitommiI closed the issue, no plans on making the parser faster or caching at the moment. Thanks for @ben.sless for digging into this 🙇#2021-07-0512:46Ben SlessI think it is worth to leave a comment on the issue and perhaps a note in the readme regarding references to uncompiled schemas#2021-07-0512:46Ben SlessAnd you're welcome 🙂 Turns out I was completely wrong regarding the transformers and satisfies?#2021-07-0512:47Ben SlessI see you left a comment with a link to the gist, so that would leave only the readme#2021-07-0513:14ikitommiif you have idea what and where to write this, please do 🙂#2021-07-0210:50ikitommi@ben.sless, great work. Saw the thread on Clojureverse about compiler optimizations. Would be great if things got automatically or magically faster. While waiting, writing performant code inside the libraries (on the hot perf path) is just being smart.#2021-07-0211:21Ben SlessThank you, I appreciate it. Blame it all on the talk you gave on ClojureTRE 2019#2021-07-0313:15ikitommitoo hot, quick poke on instrumenting functions. moving functioin wrapping code from malli.generate into malli.core (just one fn) ->`m/-instrument`, can be used to enforce function schmas. First user will be malli.instrument utilities, but could be used directly in client code too:
(def pow2
  (m/-instrument
    {:schema [:=> [:cat :int] [:int {:max 6}]]}
    (fn [x] (* x x))))

(pow2 2)
; => 4

(pow2 "2")
; =throws=> :malli.core/invalid-input {:input [:cat :int], :args ["2"], :schema [:=> [:cat :int] [:int {:max 6}]]}

(pow2 4)
; =throws=> :malli.core/invalid-output {:output [:int {:max 6}], :value 16, :args [4], :schema [:=> [:cat :int] [:int {:max 6}]]}

(pow2 4 2)
; =throws=> :malli.core/invalid-arity {:arity 2, :arities #{{:min 1, :max 1}}, :args [4 2], :input [:cat :int], :schema [:=> [:cat :int] [:int {:max 6}]]}
#2021-07-0313:17ikitommilike everything else, can be configured to behave differently. e.g. just printing out the errors (or to tap ’em):
(def multi-arity-pow
  (m/-instrument
    {:schema [:function
              [:=> [:cat :int] [:int {:max 6}]]
              [:=> [:cat :int :int] [:int {:max 6}]]]
     :wrap #{:input, :output}
     :report (fn [error props]
               (println "\n" error "\n")
               (clojure.pprint/pprint props))}
    (fn
      ([x] (* x x))
      ([x y] (* x y)))))

(multi-arity-pow 4)
;:malli.core/invalid-output
;
;{:output [:int {:max 6}],
; :value 16,
; :args [4],
; :schema [:=> [:cat :int] [:int {:max 6}]]}
; => 16

(multi-arity-pow 5 0.1)
;:malli.core/invalid-input
;
;{:input [:cat :int :int],
; :args [5 0.1],
; :schema [:=> [:cat :int :int] [:int {:max 6}]]}
;
;:malli.core/invalid-output
;
;{:output [:int {:max 6}],
; :value 0.5,
; :args [5 0.1],
; :schema [:=> [:cat :int :int] [:int {:max 6}]]}
;=> 0.5
#2021-07-0313:25ikitommioh, and the generators use this too, generator for :=> looks dead simple now:
(defn -=>-gen [schema options]
  (let [output-generator (generator (:output (m/-function-info schema)) options)]
    (gen/return (m/-instrument {:schema schema} (fn [& _] (generate output-generator options))))))
#2021-07-0314:04ikitommidunno if there is a way to emit clj-kondo annotations from schematized fns, which are not vars :thinking_face:#2021-07-0317:08respatializedSay I've got a :orn schema that I'm matching multiple values against:
[:orn [:map1 [:map {:closed true
                    :description "map1"}
              [:a :string]]]
     [:map2  [:map {:description "map2"}
              [:a :string]
              [:b {:optional true} :string]]]]
is there a way to see which of these subschemas were responsible for a given value matching the top-level form - something like m/explain but for values that do conform to the schema?
#2021-07-0317:11ikitommi@afoltzm try m/parse#2021-07-0317:16respatialized@ikitommi thanks, I knew there was likely something I was missing. my earlier examples didn't use :orn - it seems like with an unnamed :or schema only the conforming value is returned by m/parse. is there a way to return the matching schema for unnamed :or subschemas? the use case I'm thinking of here is matching values against some collection of schemas that may not necessarily be known in advance, and so can't easily be named with :orn.#2021-07-0317:17respatializedit seems like I may be able to implement something in terms of m/walk if it's not built in?#2021-07-0317:21ikitommicurrrently, no. Could add an property to :or to hint that parse should return the index of the branch or an option to parse like :malli.core/parse-indexes true. Or, you can just create :or with number-indexed branches:
(m/parse
  [:orn
   [1 :int]
   [2 :boolean]] 
  true)
; => [2 true]
#2021-07-0317:21respatializedah, gotcha. I think numbered branches is likely the easiest workaround for the time being.#2021-07-0317:23ikitommi
(defn or->orn [s]
  (m/into-schema :orn (m/properties s) (map-indexed vector (m/children s))))

(or->orn [:or :int :boolean])
; => [:orn [0 :int] [1 :boolean]]
#2021-07-0407:16ikitommiwip: malli.instrument (https://github.com/metosin/malli/pull/471)#2021-07-0507:14olyis there a way to have different map layouts, I wanted to do something like this
[:map [:coord [:or 
               [:map {:closed true}]
               [:map {:closed true} [:x int? :y int?]]]]]
{:coord {}}
{:coord {:x 1 :y 1}}
I know I can make the whole map optional but what about allowing an empty map or a map with fixed keys, I tried a few things but could not find a way.
#2021-07-0507:44alpox@oliver.marks I think this works if you use for the second :map:
[:map {:closed true} [:x int?] [:y int?]]
#2021-07-0507:47olyyeah that works for the second map, just not sure how to make it work also for an empty map I tried putting optional in various places and tried :or and a few other things oh I might have missed some brackets above but its more an an illustration of one of the methods I tried#2021-07-0507:48oly
[:map {:closed true :optional true} [:x int?] [:y int?]]
also tried this but that does not work, I know I could make the :x and :y keys optional but really I want an all or nothing method
#2021-07-0507:49alpoxI tried it with your :or above and that worked (after adding the missing brackets)#2021-07-0507:52alpox#2021-07-0507:53olythis passes ? {:coord {}} when I run it through m/valid i get missing keys or I was yesterday when I was testing 😛#2021-07-0507:53olynice, I will re check what I have maybe there was a different error which I did not spot#2021-07-0507:56oly@alpox
[:map [:coord [:or 
               [:map {:closed true}]
               [:map {:closed true}
               [:x {:optional false} int?]
               [:y {:optional false} int?]]]]]
#2021-07-0507:56olywhat about that, I think because the key do not have the optional map the :or is irrelevant anyway#2021-07-0507:58olyalthough perhaps thats it, perhaps its optional thats messing me up looking at your screenshot again I can see it fails when one of the keys are missing#2021-07-0508:09oly@alpox thanks for your rubber ducking, not an issue with malli I had 2 schmeas with similar name and was updating the wrong one 😕#2021-07-0508:10alpoxOups 🙂 np#2021-07-0508:29Ben SlessI constantly get the feeling that I'm missing a schema to talk about map schemas in terms of key-value pairs#2021-07-0508:30Ben Slessan annoying example to implement is mutual exclusion, such as a map which can have x, y and (z or u)#2021-07-0508:30Ben Slesswhere z and u have different value types, too#2021-07-0512:34ikitommiHear hear. I think something like malli-keys-relations should be there. With qualified keywords, it’s less boilerplate (but, still has):
[:or 
 [:map ::x ::y ::z]
 [:map ::x ::y ::u]]
#2021-07-0512:34ikitommispec-style would be:
[:map ::x ::y [:keys/or ::y ::u]]
#2021-07-0512:36ikitommimaybe:
[:and
 [:map ::x ::y ::z ::u]
 [:keys/or ::z ::u]]
#2021-07-0512:39ikitommi… and the mandatory JSON Schema way for this: https://json-schema.org/understanding-json-schema/reference/conditionals.html#2021-07-0512:49Ben SlessJSON Schema's BNF was one of the motivations for it#2021-07-0512:50Ben Sless> map which contains at least one ::x where ::x is [:or [:x int?] [:y float?]]#2021-07-0512:57Ben SlessIf we had it we could define JSON Schema with malli (translating the ebnf is straightforward), then define encoders and decoders to transform between the two#2021-07-0516:28ikitommiadded a video of the new function instrumention. three apis: advanced, normal users and for the lazy: https://github.com/metosin/malli/pull/471#2021-07-0516:28ikitommicomments welcome#2021-07-0603:40emccueIs there a good thing to use for "I make no assumptions about what this is"#2021-07-0603:41emccuelike any?, but for say something you can pass to next.jdbc#2021-07-0603:41emccueand won't be generatable#2021-07-0603:46emccuewriting a spec for what would count as a "db" has been confusing me, so i want a give up hatch for a second#2021-07-0609:37ikitommiso, :any is not enough? You could say [:any {:gen/elements ["whatever"]}] to generate what you like.#2021-07-0613:11emccueI guess it is with the empty gen elements.#2021-07-0612:15ikitommimiu/collect! with pluggable collector. anyone interested in doing fspec -> malli collector?#2021-07-0612:18ikitommithe source:
(defn -accept-default [v]
  (let [{:keys [ns name malli/schema] :as meta} (meta v)]
    (when schema (m/-register-function-schema! (-> ns str symbol) name (miu/-unlift-keys meta "malli")))))

(defn collect! [{:keys [ns accept] :or {accept -accept-default}}]
  (doseq [[_ v] (ns-publics ns)] (accept v)))
#2021-07-0612:46ikitommiIdea: an external tool that infers the schemas from vars and adds the needed schema meta into vars (rewrites the source, I recall there are tools for that already, edamame?).#2021-07-0615:10robert-stuttafordi use edamame on the edn that i give to malli, but only because i have the exceptions edamame gives me to tell the user where they typed something wrong when they malform edn#2021-07-0622:06borkdude> rewrites the source This would be rewrite-clj#2021-07-0813:30robert-stuttaforduse that too 😄#2021-07-0615:08robert-stuttafordMalli is lovely.#2021-07-0619:06ikitommigenerated responses from function schema definitions:#2021-07-0622:40emccue@robert-stuttaford I'll add to that - the malli syntax is solid even without an impl. We don't use malli at work yet, but we have already started using its syntax for describing data in (comment ...) blocks.#2021-07-0813:26robert-stuttafordthat's pretty cool @emccue! it has the same property as hiccup - ultra consistent data optimised for human reader#2021-07-0910:12jeroenvandijkIf I want to validate if something is a function, it seems I can’t do (m/validate fn? (fn [])), but I can do (m/validate ifn? (fn [])) or (m/validate [:=> [:cat] any?] (fn [])) . Is there an easy way to look up what predicates are supported?#2021-07-0910:12jeroenvandijkFor now it seems the source is the quickest way? I’m looking into the source now to check it https://github.com/metosin/malli/blob/50c6cf6f6438401d6de2c46a9542f4c10da76a86/src/malli/core.cljc#L1750#2021-07-0910:36alpoxThere is the section https://github.com/metosin/malli#mallicorepredicate-schemas in the docs but it appears not to list ifn? :thinking_face:#2021-07-0910:42jeroenvandijkyeah I found this through the README https://github.com/metosin/malli#fn-schemas I’m guessing omission in the docs#2021-07-0916:46ikitommicould you @U0FT7SRLP update the docs?#2021-07-0916:48ikitommiwhen instrumentation is done, there could be a separate doc for function schemas. Lot's of different aspects (validation, generative testing, instrumentation, linting, ...) to cover.#2021-07-1015:42ikitommioh, fn? is missing, will add that
#2021-07-0916:48ikitommiwhen instrumentation is done, there could be a separate doc for function schemas. Lot's of different aspects (validation, generative testing, instrumentation, linting, ...) to cover.#2021-07-1015:41gregHi, I get my feet wet in malli for the first time. I watched a couple of videos, I read some articules and scrolled the malli readme, and I;m still a bit confused about transformers. I'm trying to create a bigdec? pred and transformator for it, so for given data structure I could coerce selected values from string and number to bigdec. I looked for some examples on the internet. There are some examples of using pre-defined transformers but no examples of custom transformers. I mean, I haven't checked the code of Malli, I guess, that is the ultimate source of info. Questions I'm trying to answer now: • what are the guidelines for writing transformers? • how to define custom predictor (in similar fashion to string?)? • how to marry a custom predictor with a custom transformer (so when coercing a data structure, the transformer can detect properties to coerce for my custom predictor) Any hints, links, would be very helpful. Thank you 🙇#2021-07-1020:54ikitommigood place to start could be this: https://github.com/metosin/malli#simple-schema#2021-07-1020:54ikitommihas a custom type, with a custom transformer, generator, json-schema mapping & humanized error dedinition#2021-07-1115:00gregI had to miss this part. Thank you#2021-07-1115:07gregI used to use Clojure Spec before, but I have never used Plumatic Schema before. I know it might sound ridiculously silly, but I haven't matched in my couscous mind the word 'schema' with a data shape definition. I mean, it makes total sense now, just I so get used to the word 'spec'. I don't know how I was watching this videos and reading these posts without that basic vocabulary understanding 😅#2021-07-1205:06datranWhat's the best way to update the properties of a map entry? I've got a schema like [:map [:a :string] [:b :string]], and I want to transform it into something like [:map [:a :string] [:b {:my-date :foo} :string]]. mu/update-properties only works for the :map-level, so I tried combining it with mu/update and giving a path into the map, but that also doesn't seem to be what I want. Is there a simpler way to achieve this that I'm overlooking?#2021-07-1315:42ikitommiYou can walk the children or wait for https://github.com/metosin/malli/pull/466 to be merged#2021-07-1315:45ikitommi
(defn map-children [f schema]
  (m/-into-schema (m/-parent schema) (m/-properties schema) (map f children) (m/-options schema)))
#2021-07-1315:46ikitommithat's the idiomatic way to create a copy of a schema with some changes#2021-07-1315:45ikitommi
(defn map-children [f schema]
  (m/-into-schema (m/-parent schema) (m/-properties schema) (map f children) (m/-options schema)))
#2021-07-1310:25gregIs it possible to define a default value for a map entry in terms of other map entry, e.g. there are two keys in a map: primary and secondary. Secondary if missing is set based on primary one.#2021-07-1310:26gregSchema expressing it could look like this:
[:map
 [:primary string?]
 [:secondary {:default-fn '(fn [m] (:primary m))} string?]]
#2021-07-1311:02ikitommiout-of-the-box no, but you can implement this is user space, using a custom transformer. I recommend reading the source code of mt/default-value-transformer. My guess is that you should register the transformer for the map-level, peek the entrys and given there is :default-fn , call it with the map value. Should be streightforward. Could be a nice example into docs/tips.md...#2021-07-1311:14gregThank you!#2021-07-1617:22gregI had a look today in the default value transformer, and based on what you wrote I've made an example of a default value exercised by given function instead of default value (in the current impl). I modified the default-value-transformer to do that. But at the end I realised that I could handle this case:
[:map
 [:primary string?]
 [:secondary {:default-fn '(fn [m] (:primary m))} string?]]
but I wouldn't be able to handle this one:
[:map
 [:primary string?]
 [:nested 
  [:map 
   [:secondary {:default-fn '(fn [m] (:primary m))} string?]]
because the given fn receives only current map/submap not the root map? How would you tackle that kind of transformation?
#2021-07-1617:34ikitommisimplest would be just to add a normal transformer into the top-map, which would use normal clojure to transform the whole nested map.#2021-07-1617:38ikitommiyou could make it declarative thou, but it's not simple: 1. add the transformer to the top-level 2. add the declarations to any children 3. use the :compile hook to 1 to get access to the full nested schema 4. within it, walk the schema and collect all the declarations and the paths, create a function to transform the child values 5. on transformation, the "compiled" transformation is applied at the top, knows already what to do, is fast#2021-07-1617:42ikitommi(actually, it is simple) 😉#2021-07-1618:23greg> simplest would be just to add a normal transformer into the top-map, which would use normal clojure to transform the whole nested map. you mean, without Malli? I know, that would be the simplest, but I'm trying to understand malli transformers thoroughly and this problem looks like a good one to understand how Malli transformers works. > actually, it is simple) 😉 I believe you it is simple, but at my stage of understanding malli transformers I'm reading each your response 10 times and it is not enough 🙂 I mean i understand briefly some of the bits already, based on the code and docs, but it it still not enough to understand some of the terminology you use. TBH the first three steps are not really clear to me. What do you mean by "add the transformer to the top-level" "add the declarations to any children" "use the `:compile` hook to 1 to get access to the full nested schema" - I briefly know how the compilation works (preprocessing of the schema and passing discovered data within closured fn into interceptor), but I don't understand the phrase "hook to 1" --- Also, within (transformer) I noticed that interceptors are created from transformers (from :decoders or :encoders) and default (from :default-decoder or :default-encoder). default-value-transformer uses both. I mean, I look at the code and I don't understand what is it about. Looks like something obvious once you know it. Can you tell what is it about?#2021-07-2714:28gregI got back to the topic, trying to find all answers to my question. @U055NJ5CC would you be so kind to read it and correct or complement if something is missing in the notes below. If the notes are correct, I could polish them a bit, maybe add examples, and add it to the Malli docs if you wish. There is a hierarchy of hooks that Malli handle when transforming values (in this order of priority): • from schema properties: :decode/<name> e.g. :decode/math - provides a transforming fn for a schema enclosing it (to enable, transformer specifying the given name need to be applied e.g. (mt/transformer {:name :math}) • from schema type properties: :decode/<name> e.g. :decode/math - provides a transforming fn for a schema type enclosing it (to enable same as above) • from transformer definition: :decoders, :encoders - provides a map of (schema -> transforming fn), so for the every such schema in the given value's tree, applies the given fn • from transformer definition: :default-decoder, :default-encoder - provides a last resort transforming fn, so if none of the above transforming fn hooks do not apply, it takes this one. This fn just takes the whole schema, so it might be called a top level hook. Moreover: • :enter/`:leave` refers to the interceptor stages, so we can provide separate transforming fn, they hold directly final functor modifying the the value. The map holding :enter/:leave can be provided by either compiling function or directly. • compiling is the technique exercising values closure. To use it, instead of transforming function, a map {:compile <compiling-fn>} needs to be given. Compiling fun it is a fn taking schema and schema options, and returning a transforming fn. That way, processing of schema specifics can be done once at the time of constructing transformer. #2021-07-1312:08ikitommiWIP: function schema guide. Wanted to push all aspects into single page, so that it’s easier to read what is possible and how the things stack up. Is that any good? something missing feature-wise? have time to polish things this week, comments most welcome: https://github.com/metosin/malli/blob/08111040566e2d39ecff62dfccdcb834f7a23140/docs/function-schemas.md#2021-07-1312:09ikitommiPR is here for comments & fixes: https://github.com/metosin/malli/pull/471#2021-07-1312:30gregSo actually orchestring project could be done in terms of m/=>, (mi/start!) and (mi/stop!). Am I correct?#2021-07-1312:39ikitommiyes.#2021-07-1312:21gregJust a note about orchestrating projects using Malli schemas. I wanted to setup orchestration using malli in a way similar I used to do it with Orchestra. I checked out https://github.com/teknql/aave, https://github.com/setzer22/malli-instrument and https://github.com/CrypticButter/snoop. Snoop: I've found it the best one for that purpose. Works fine with functions evaluated in REPL, readable error messages. A bit odd way of enabling it (by using JVM flag), on the other hand once set in dev and test aliases, it just works, for REPL and tests. Also 'https://github.com/CrypticButter/snoop#inside-the-prepost-map' syntax for adding function schemas is great. malli-instrument: similar syntax to spec (call instrument-all to instrument), but there are problems with orchestration when you modify and evaluate continuously schemas and functions within REPL. aave: I have had several looks at this library, and its README and TBH I have no idea how to use it. I guess I'm missing some relevant piece of information regarding instrumentation with Malli.#2021-07-2303:11Lucy WangI also find snoop very handy. To keep the schema inline with the arg name is a huge win
(>defn xy*2
    [(x :int) (y :int)]
    [=> :int]
    (* 2 x y))
#2021-07-2311:28gregAt first it was weird to me that Snoop supports so many options for providing inline schemas, but at the end I used 3 variants and at the 3rd time I found my flavour (just plain [:input :output]). So I'm kinda glad that Snoop doesn't enforce to use one particular syntax 🙂 I'm surprised that not that many people use orchestration. Seriously, it is one of the killer features of Clojure. And with Malli and its data-driven approach, it is so pleasant and concise to write and modify specs/schemas. I'm aspiring to deliver to market an accounting software and when refactoring some of the code that does calculations or refactoring, the code orchestration is such a helper, always watching and keeping that invariants are valid.#2021-07-2404:15Lucy Wang@U023TQF5FM3 By "orchestration" do you mean https://github.com/jeaye/orchestra ?#2021-07-2404:18Lucy Wang> at the 3rd time I found my flavour (just plain `[:input :output]`). Which flavor are you talking about?#2021-07-2416:47gregI prefer providing schemas in a two elements vector following the params vector.
(>defn add [x y]
  [[:cat int? int?] int?]
  ...)
I found it the nicest to read 🙂 Here is an example: https://github.com/CrypticButter/snoop#more-convenient-notations-that-work-when-using-defn
#2021-07-2416:48greg> By "orchestration" do you mean https://github.com/jeaye/orchestra ? I use the "orchestration" term for instrumentation of selected methods in a project, so you provide specs/preds/invariants and the tool using instrumentation watches the instrumented functions, either using Orchestra & spec or Snoop & Malli, within your REPL and tests. I think the distinction between these two, is that instrumentation is a technique, while orchestration is a practice that leverages instrumentation.#2021-07-1312:36ikitommi@grzegorzrynkowski_clo did you check the guide I just posted? There will be a malli.instrument ns, with common utilities for spec/orchestra-style usage. It's all built on existing function (var) registry, so no new macros/defn-syntax. It would be great if all defn-wrapping libs could use malli.instrument internally, would keep thing more coherent. Not sure if that is possible thou. All existing libs (aave, Snoop, malli.instrument) are great, just that the core instrumentation belong to the core. Better defn wrappers more welcomed outside.#2021-07-1312:45gregI've seen it, just was looking for some ready to go solution right now. When malli.instrument ns comes, most probably I will reevaluate my setup 🙂#2021-07-1313:19gregIs there a function to validate schemas' data structures? I was playing with Malli and transformers and I was trying to figure out what is wrong with my custom schema. I tired to coerce data using string-transformer and it didn't work. It turned out the schema of my schema was wrong. Here is an example showing my problem:
(def Order
  [:map
   [:qty number?]
   [:price string? float?]])

(comment
 (m/validate Order {:qty "1" :price "1.1"}) ; => false
 (m/validate Order {:qty 1 :price "1.1"}) ; => true
 (m/decode Order {:qty "1" :price "1.1"} mt/string-transformer) ; => {:qty 1.0, :price "1.1"}, but I expected {:qty 1.0, :price 1.1}
 (->> (m/decode Order {:qty "1" :price "1.1"} mt/string-transformer)
      (m/validate Order))) ; => true
Schema [:price string? float?] is wrong. Unintentionally I forgot to remove string? . Still it is fairly small example, but within complex examples might be difficult to spot it. Would be nice to have a method checking schemas data structures. Is there something like that?
#2021-07-1313:44emccueSounds like you want a schema for your schemas#2021-07-1313:57ikitommisomeone needs to define those for all schemas. Malli supports them already#2021-07-1617:22gregI had a look today in the default value transformer, and based on what you wrote I've made an example of a default value exercised by given function instead of default value (in the current impl). I modified the default-value-transformer to do that. But at the end I realised that I could handle this case:
[:map
 [:primary string?]
 [:secondary {:default-fn '(fn [m] (:primary m))} string?]]
but I wouldn't be able to handle this one:
[:map
 [:primary string?]
 [:nested 
  [:map 
   [:secondary {:default-fn '(fn [m] (:primary m))} string?]]
because the given fn receives only current map/submap not the root map? How would you tackle that kind of transformation?
#2021-07-2303:11Lucy WangI also find snoop very handy. To keep the schema inline with the arg name is a huge win
(>defn xy*2
    [(x :int) (y :int)]
    [=> :int]
    (* 2 x y))
#2021-07-1416:35wcalderipeHey folks, I'm trying to use Malli to parse keys and values of a map to something else but I'm having a bit of difficulty understanding how and what function to use. I've tried m/decode and m/parse so far but not with much success. How can I achieve a transformation like the one below using Malli?
;; input
  {:foobar "baz" 
   :qux    "999"}

;; output
  {:foo :baz ;; Changes the key from :foobar to :foo
   :qux 999  ;; Coerces the value from string to integer 
   }
For context, the use case I have in mind is parsing the response of an API to something else. I'm sharing it because maybe there's a better way to use Malli for schema checking of services we don't control plus parsing the data to something else.
#2021-07-1417:07ikitommi@wcalderipe maybe:
(def decode
  (m/decoder
    [:map
     [:foo :keyword]
     [:qux :int]]
    (mt/transformer
      (mt/key-transformer {:decode #(get {:foobar :foo} % %)})
      (mt/string-transformer))))

(decode
  {:foobar "baz"
   :qux "999"})
; => {:foo :baz, :qux 999}
#2021-07-1417:20wcalderipeGot it.. thanks for taking the time to reply 🙌 I have another question between these lines. Can I reuse the same schema for validating the API response and do the whole transformation? Let's say we're integrating a Todo API and we need to do some transformation on their response.
;; Todo API schema as it is
(def todo-api
  [:map
   [:name [:string {:min 1}]]
   [:done [:boolean]]
   [:create_at [:string]]])

(malli/validate todo-api {:name      "Ask for help at #malli channel"
                          :done      true
                          :create_at "1615007456"})
;; => true
1. Validate if responses are following the schema 2. Parse the response to something else as you've shown
;; Output
{:name      "Ask for help at #malli channel"
 :done      true
 :timestamp 1615007456} ;; Rename the key and transform the value
Do I need two schemas 1 for the input and 1 for the output in a scenario like this? :thinking_face:
#2021-07-1510:14gregMy understanding is that Malli encourage to transform first and validate next. For example, let's say data you are receiving are invalid. You decode, validate it. It turned out something is wrong you might want to return an original piece of data by transforming it back (using m/encode) and adding some info what's wrong. If the response has a different schema by its identity, make sense to have a different schema. Disclaimer, I'm not an expert in Malli 🙂#2021-07-1510:24gregSo should you have one schema or two, is question are these data same thing or two different things. And answer is not always clear. E.g. In layered architectures, like MVP, depends on implementation you might have several schemas (representation) of the same data.#2021-07-1512:18ikitommiif the web-facing values can be mapped automatically to internal values, you should be able to do the same transformation for the schemas too, e.g. define just one and generate the other. Reason for wanting to have two schemas could be apidocs: they should be generated from “web-facing” schemas, not the internal model.#2021-07-1512:18ikitommirelated: https://www.metosin.fi/blog/malli/#schema-transformation#2021-07-1512:20ikitomminot sure if there are all the needed helpers in malli.util , but easy to add if something is missing.#2021-07-1606:00wcalderipeThanks for taking the time @UBVL1LR5F > E.g. In layered architectures, like MVP, depends on implementation you might have several schemas (representation) of the same data. I think the problem I have in mind, the data means the same thing but it has a different shape inside the boundaries of my app. Malli will be used near to the edges of the app into something like an anti-corruption layer to translate an external concept to an internal one. That's renaming some keys and coerce some of the values too.#2021-07-1613:22gregIf the Malli resides only on the one side of the ACL architecture, on the internal one, I think you could work with one schema. If both would use Malli, then I would go with two. Going back to you question > Can I reuse the same schema for validating the API response and do the whole transformation? I think yes, you could use one. You received response from external (behind ACL) system, you transform (including the keys) and then validate.#2021-07-1613:24gregIf you want validate first for some reason, you might need two.#2021-07-1708:24wcalderipeThanks for sharing @U055NJ5CC My doubt is more a question of design than which functions to use. @UBVL1LR5F got it... > If you want validate first for some reason, you might need two. Yeah, I think I'll end up with two instead of one because I was thinking of doing validations as well in development mode. Thanks a bunch for the help!#2021-07-1512:45ikitommimerged malli.instrument PR into master. will be released as part of 0.6.0 later. test reports, feedback, improvements welcome! the new function guide is here: https://github.com/metosin/malli/blob/master/docs/function-schemas.md 😎#2021-07-1609:46hansbuggeHi, I'm experiencing something funny with humanize in Malli 0.5.1. When the wrapper function returns a string, then the :malli/error key gets an entry for each error, but when it returns anything but a string then there will be only one error under :malli/error.
(-> (m/explain [:and
                [:fn (constantly false)]
                [:fn (constantly false)]]
               {:a :map})
    (me/humanize {:wrap (constantly "a string")}))
;; => #:malli{:error ["a string" "a string"]}

(-> (m/explain [:and
                [:fn (constantly false)]
                [:fn (constantly false)]]
               {:a :map})
    (me/humanize {:wrap (constantly :not-a-string)}))
;; => #:malli{:error [:not-a-string]}
#2021-07-1609:48hansbuggeAnd I'm having a hard time understanding why from looking at the source. Am I right that this is a bug?#2021-07-1609:56hansbuggeIt seems to be the form (if (-just-error? v) (into (vec e) v) v) in malli.error/-put, where -just-error? only returns true if the value is exactly a vector with one string#2021-07-1610:01hansbuggeMy problem is that we're using explainers and humanize to generate structured form validation output, where the wrapper passed to humanize generates a map with the error message and a couple of other fields. But this means we get at most one error in :malli/error.#2021-07-1613:48ikitommijust wrote a issue based on old PR to make the :and error handling more robust, e.g. take the first error form and accumulate to that. Not sure if that would help here, but coming anyway: https://github.com/metosin/malli/issues/476#2021-07-1615:45ikitomminice to see how things click while developing malli. • in spec1, s/and flows conformed values, which is IMO not intutive, but needed for special cases (e.g. parsing sequential input), source of much confusion • in spec1, s/cat forced things to be named, need you it or not • in spec2, there might be a non-conforming variant and', most likely no cat' thou. with malli, the simple syntax is the default and you can detail it (named branches, parsed childs) if needed. Spec has amazing ideas and personally is a big source of inspiration. The Malli evolution: Given:
(defn distance [min max] 
  (- max min)
Simplest description:
;; int int -> int
[:=> [:cat :int :int] :int]
Adding names:
;; min:int max:int -> int
[:=> [:catn [:min :int] [:max :int]] :int]
Adding constraints on input (with new parsed utility schema):
;; min:int max:int [min < max] -> int
[:=>
 [:and
  [:catn
   [:min :int]
   [:max :int]]
  ;; here we want to used parsed values instead of orginal sequence
  [:parsed
   [:fn
    {:error/message "min should be lass than max"}
    (fn [{:keys [min max]}] (< min max))]]]
 :int]
#2021-07-1617:39gregHaha, I've spend a couple of hours trying to understand the logic behind it recursive traversal of transformers until I discovered it is at the Schema implementation, in the transformer- fn 😄#2021-07-1705:07mike_ananev@ikitommi , hi! I've tried to run static type checking using FileWatchers + clj-kondo. Can't see any effect for plus1. clj-kondo works well. malli version 0.6.0-snapshot#2021-07-1709:06ikitommidid you add the clj-kondo configs for malli? Not sure if that could be automatic in the future? Maybe @U04V15CAJ knows? https://github.com/metosin/malli#clj-kondo#2021-07-1709:11ikitommie.g.
✗ cat .clj-kondo/config.edn
{:config-paths ["configs/malli"]}
#2021-07-1709:42borkdudeconfigs are never opted into automatically#2021-07-1709:43borkdudeby design, you should opt into those manually, for safety and consistency reasons#2021-07-1710:07ikitommiSounds right. But is there a default directory I could write the config file to, without breaking anything?#2021-07-1710:08borkdudeit's best to write into a directory with a fully qualified name like com.metosin.malli or so#2021-07-1710:08borkdudeto avoid conflicts#2021-07-1710:08borkdudeor just the domain name of your library#2021-07-1710:16ikitommigood idea. but despite being fully qualified, users need to enable that config in their projects, right? no option to "trust 'em all"?#2021-07-1710:17borkdude> by design, you should opt into those manually, for safety and consistency reasons#2021-07-1710:17borkdude;)#2021-07-1710:18borkdudeyou should just add that dir to your config paths, that's it#2021-07-1710:18borkdudeany tool can dump any config into the dir, but it's not always something a user wants to opt into#2021-07-1710:19mike_ananev@ikitommi you was right! I added {:config-paths ["configs/malli"]} to .clj-kondo/config.edn and now clj-kondo works well with malli static type checking#2021-07-1710:19borkdudeyesterday I had such a report where clojure-lsp dumped a config into the dir, but a user did not want to have this config#2021-07-1705:07mike_ananev#2021-07-1810:24Vincent CantinI have a schema that parses a value correctly but then the unparse operation returns :malli.core/invalid. Is it a known problem for some kind of schema or should I post a bug report?#2021-07-1811:39Vincent CantinTo reproduce:
(let [my-schema (m/schema
                  [:and
                   list?
                   [:catn
                    [:wrapper [:= 'value]]
                    [:wrapped int?]]])]
  (->> (m/parse my-schema '(value 5))
       (m/unparse my-schema)))
#2021-07-1811:40Vincent Cantinm/unparse works if I remove list? from the schema, but then it's not the schema I need.#2021-07-1811:54Vincent CantinIs there an option in catn to specify the list of container of the sequence?#2021-07-1812:57ikitommicurrently no, but definetely it should.#2021-07-1812:57ikitommimaybe something like?
[:cat {:type :vector} :int :int]
#2021-07-1812:59Vincent Cantinyes, but the name "type" is not descriptive enough as it relates to the container's type and not the sequence itself.#2021-07-1813:00Vincent CantinIn minimallist, I named it "coll-type" https://github.com/green-coder/minimallist/blob/all-work-and-no-play/src/minimallist/helper.cljc#L72-L76#2021-07-1813:02Vincent CantinRight now, I am trying to port my model from minimallist to malli in the Vrac project, so I have less things to maintain and it's better for the users if they face a mainstream library like Malli.#2021-07-1813:14ikitommihave you checked the bundle size? malli starts from ~2kb (just the minimal working set of schemas), but is quite big if all schemas are used (40kb?). with all the bells & whistles, it can be >100kb.#2021-07-1813:15ikitommiI’m not sure how :type and :coll-type are that different, but something like that would be good.#2021-07-1813:17Vincent Cantin:type will be fine#2021-07-1813:28Vincent Cantinthe bundle size won't be a problem, I can parse things offline.#2021-07-1818:16Vincent Cantin:tuple would also need to have the {:type :list} option.#2021-07-1810:30Vincent CantinWhile chaining parse and unparse, I realized that the API would be more friendly to -> if the parameter order of ?schema and value were exchanged.#2021-07-1813:02ikitommidid a quick run on improving performance of collection transformations, got easy x5 for CLJ in simple test:
(let [schema [:map
              [:id :string]
              [:type :keyword]
              [:address
               [:map
                [:street :string]
                [:lonlat [:tuple :double :double]]]]]
      decode (m/decoder schema (mt/json-transformer))
      json {:id "pulla"
            :type "herkku"
            :address {:street "hämeenkatu 14"
                      :lonlat [61 23.7644223]}}]
  ;; 920ns => 160ns
  (cc/quick-bench
    (decode json)))
#2021-07-1813:05ikitommialso, coercion (transform + validate) is 5x faster with that data compared to Plumatic Schema. Which is kinda nice as have considered (the awesome) Plumatic as a performance reference.#2021-07-1813:05ikitommimerged in master.#2021-07-1813:05ikitommihttps://github.com/metosin/malli/pull/478#2021-07-1813:14Ben SlessI think you can close over the iteration completely#2021-07-1813:42Ben Sless@U055NJ5CC you can gain at least 10% and probably gain some mechanical sympathy by closing over the transforms:
(map
 (fn [[k v]]
   (fn [^Associative x]
     (if-let [xe (.entryAt x k)]
       (.assoc x k (v (.val xe)))
       x)))
 ts)
then reduce over them with -comp
#2021-07-1813:47Ben SlessI can PR this if you'd like#2021-07-1818:58ikitommilooks good. But, I thing the varargs version of -comp is actually slow as it uses the sequence abstraction, could be rewritten to use iterator? PR (and perf test before & after) most welcome.#2021-07-1819:01Ben SlessI unrolled it to 16 args#2021-07-1819:21ikitommi👍#2021-07-1819:22ikitommialready this is much faster that the current:
(defn -comp
  ([] identity)
  ([f] f)
  ([f g] (fn [x] (f (g x))))
  ([f g h] (fn [x] (f (g (h x)))))
  ([f1 f2 f3 & fs]
   (let [fs (into [f1 f2 f3] fs)]
     (fn [x] (let [i (.iterator ^Iterable fs)]
               (loop [x x] (if (.hasNext i) (recur ((.next i) x)) x)))))))
#2021-07-1819:45Ben SlessI'll send an organized PR tomorrow, today is getting late#2021-07-1907:41Ben SlessI wonder if there's an optimal maximum arity#2021-07-1813:05Ben Sless👀#2021-07-1813:07ikitommiMy initial thought was that having transform and validation as separate steps can actually make it faster - as the created transformation chain is usually small enough to fit into the JVM inlining budget - while validation is always “complete” and generated more code. Having those in one sweep means more code. Haven’t looked at the perf profile, so just quessing.#2021-07-1813:10Ben SlessI was just thinking about this recently - would interleaving transform and validation be faster? Since it involves lots of iteration and allocation, we can't necessarily assume JIT friendly code would be faster than a single pass over two passes#2021-07-1813:08Ben Sless> entryAt 😄#2021-07-1813:09ikitommi#2021-07-1813:09ikitommithe initial code was using reduce for all, which does a lot of things.#2021-07-1813:10ikitommistack size from 36 -> 6 (on malli side).#2021-07-1813:11ikitommibut thanks @ben.sless for the validation perf, mostly same optimizations for transformers 🙇#2021-07-1813:12Ben SlessI thought it looked familiar :thinking_face:#2021-07-1813:12Ben Sless😄#2021-07-1813:17ikitommireduce-kv should be fast, but for some reason, it’s not always used instead of reduce. Recall there was a related bug in clojure for this. writing by hand is always fast 🙂#2021-07-1813:27Ben SlessHave you had a chance to look at my proposal on https://github.com/metosin/malli/issues/474?#2021-07-1813:46Ben SlessI have no idea how to compile my proposal to something efficient, though I assume it is possible and someone clever like nilern could do it#2021-07-1820:07Vincent CantinIt is possible in Malli to write a schema which represent Clojure's destructurations of a map? For example, something which would parse this kind of data:
{a :a
 b :b
 :keys [c d]
 :e/keys [f g]
 :as h}
#2021-07-1907:49ikitommiI don't think there is. Does spec have something for this?#2021-07-1907:49ikitommiand what would be the expected parse result?#2021-07-1907:51Vincent CantinI don't know if spec can do something like that. In minimallist, I approach this problem by having map-of taking a schema of a pair [key value], so I am able to use :alt on the pair to valid multiple different cases where the key and the value are correlated.#2021-07-1907:52ikitommimaybe m/parse could have a custom user-defined property to allow users to give the parse & unparse functions for the given schema.#2021-07-1907:53Vincent CantinAn escape hatch, yes it would work, but the hardship would be on the user.#2021-07-1907:53ikitommicould you write a minimallist sample for that case? Sounds interesting#2021-07-1907:53Vincent CantinHere it is: https://github.com/green-coder/vrac/blob/diy-furry/src/vrac/model.cljc#L25-L30#2021-07-1907:56Vincent CantinThe escape hatch may not work well for the user in the case the model is using recursion, like in this example. That would become Malli -> fn -> Malli -> fn ...#2021-07-1820:15respatializedHi all! I've been working for a while on fabricate, a static website generator that takes advantage of malli in order to both provide a model for semantically valid HTML/Hiccup forms and to define its own order of operations. I wrote a post about my experience using malli schemas to define a finite state machine that dispatches functions on the basis of the schemas matched by the data passed in to them, which I have taken to calling "finite schema machines." https://fabricate-site.github.io/fabricate/finite-schema-machines.html I'd be really interested in what other users of malli think about this concept. It's highly experimental, but I've already found it quite interesting and useful.#2021-07-1909:30Ben SlessThis looks very interesting and I've had FSMs on my mind recently. Will need to give a deeper look#2021-07-1910:04Ben SlessWonder how this relates to dependent types#2021-07-1912:50respatializedI've never worked with dependent types, but my impression of malli is that you get a lot of the benefits of dependent types without a static type system. that said, when I think about how to validate/check these FSM definitions for cycles or non-termination at definition/compile time, I wonder how close I'm actually getting to reinventing a type system.#2021-07-1907:47ikitommi@afoltzm looks awesome!#2021-07-1920:20mike_ananev@ikitommi, do you have any plans to release malli 0.6.0 soon? Malli 0.6.0-snapshot has very exciting functionality about function schemas and instrumentation.#2021-07-2009:13ikitommino plans, but for the instrumentation part, the latest immutable SNAPSHOT version should be stable'ish (or depend on latest commit via deps)#2021-07-2009:13ikitommino plans, but for the instrumentation part, the latest immutable SNAPSHOT version should be stable'ish (or depend on latest commit via deps)#2021-07-2003:03escherizeIs there a malli schema for the clojure.core/defn form?#2021-07-2004:50mike_ananev@escherize https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defn-schemas-via-metadata#2021-07-2005:45escherizethanks for your response @mike1452. What I am looking for is a malli schema that will say “that clojure.core/defn form you have is good and not malformed” or not.#2021-07-2005:47escherizeI think the schema should match these, but there may be more:
['(defn f [a] 1)
 '(defn g [a] (+ a a))
 '(defn ^:ok g "hi" [a] (+ a a))
 '(defn ^:ok g "hi" ([a] (+ a a)))
 '(defn ^:ok g "hi" ([a] 1) ([a b] 2))]
#2021-07-2006:00escherizeI thought I was getting close, but can’t quite get it for all cases. Maybe someone here can figure it out
(def DefnSchema
  [:cat
   [:enum 'defn '-defn]
   symbol?
   [:? string?]
   [:alt
    [:cat [:vector symbol?] [:* any?]]
    [:* [:sequential [:cat [:vector symbol?] [:* any?]]]]]])

(mapv #(m/explain DefnSchema %)
      ['(defn f [a] 1) ;;<- good
       '(defn g [b] (+ b b))  ;;<- good
       '(defn ^:ok g "hi" [c] (+ c c))  ;;<- good
       '(defn ^:ok g "hi" ([d] (+ d d))) ;;<- fail
       '(defn ^:ok g "hi" ([e] 1) ([e e] 2))]);;<- fail
#2021-07-2011:10ikitommim/explain might hint why it's not right#2021-07-2011:10ikitommioh, you had that.#2021-07-2011:11ikitommiany hint about why they failed? (not near computer myself)#2021-07-2013:47danielnealIs there already a transformer that will work with query parameters - specifically to convert single values into vectors where the schema specifies a vector. i.e.
(malli.core/decode [:map
                    [:a [:vector int?]]]
                   {:a "1"}
                   some-transformer) => {:a [1]} (as opposed to {:a "1"})

(malli.core/decode [:map
                    [:a [:vector int?]]]
                   {:a ["1" "2"]}
                   some-transformer) => {:a [1 2]}
This seems to work - but is it right?
(defn collection-transformer []
  (malli.transform/transformer
    {:decoders
     {:vector
      {:compile (fn [schema _]
                  (fn [x]
                    (if (vector? x) x [x])))}}}))
#2021-07-2013:52ikitommi@escherize it should be:
(def DefnSchema
  [:cat
   [:enum 'defn '-defn]
   symbol?
   [:? string?]
   [:alt
    [:cat [:vector symbol?] [:* any?]]
    [:+ [:schema [:cat [:vector symbol?] [:* any?]]]]]])
#2021-07-2013:53ikitommie.g. not sequence :+ of sequence :sequence of sequences :cat, just sequence (`:+`) of sequences :cat.#2021-07-2013:58ikitommi@danieleneal maybe something like:
(m/decode
  [:map
   [:a [:vector int?]]]
  {:a "1"}
  (mt/transformer
    (mt/transformer
      {:decoders {:vector (fn [x] (if (string? x) [x] x))}})
    (mt/string-transformer)))
; => {:a [1]}
#2021-07-2014:06danielnealah cool, thanks, looks like I’m on the right lines 🙂#2021-07-2014:09ikitommiyou can use :compile if you want to access the schema ahead of time, like reading the separator per schma:
(def decode
  (m/decoder
    [:map
     [:a [:vector {:separator ";"} int?]]]
    (mt/transformer
      (mt/transformer
        {:decoders
         {:vector
          {:compile (fn [schema _]
                      (let [separator (-> schema m/properties :separator (or ","))]
                        (fn [x]
                          (cond
                            (not (string? x)) x
                            (str/includes? x separator) (into [] (.split ^String x ^String separator))
                            :else [x]))))}}})
      (mt/string-transformer))))

(decode {:a "2"})
; => {:a [2]}

(decode {:a "1;2"})
; => {:a [1 2]}
#2021-07-2014:25ikitommi@danieleneal https://github.com/metosin/malli/blob/master/docs/tips.md#decoding-collections#2021-07-2014:35danielnealnice!! Thanks :))))#2021-07-2014:29kennyI'm curious if Malli has considered allowing the default registry to be set in a less invasive way (i.e., not through jvm props)? The requirement to set a jvm prop touches many places in a large application and, I imagine, will cause future developer confusion by requiring every repl to be launched with that prop.#2021-07-2014:43ikitommitotally agree. the current way to enable custom registry is too much work and one can always define an immutable registry for the cases (multi-tenant env) when it matters. Just not sure what would be a best possible compromise between simple & easy here. Imperative programming with global state is not good either. Ideas?#2021-07-2014:44ikitommithere was a discussion somewhere some time ago..#2021-07-2014:46ikitommi1. immutable by default, swapping needs a custom jvm/compiler option 2. mutable registry by default, spec-like 3. immutable by default, but mr/set-default-registry! available without jvm options 4. something else#2021-07-2015:10kennyWe are very early in our usage of Malli, coming from a large use of Spec, so take anything I say with that grain of salt. I quite like the default mutable registry. Over the years, we have built up a large library of domain specs that are used all over the place. It's handy to be able to simply reference these specs by their keyword name. Of course, Malli may encourage different conventions (e.g., just write functions returning custom schema). I started down the path of creating our own immutable registry, but started to feel pain when I needed to pass my registry to every single Malli api call.#2021-07-2015:25mike_ananev@ikitommi I found breaking changes between 0.5.1 and 0.6.0-snapshot in a function m/validate In 0.5.1, function m/validate returns true if data corresponds to spec. In 0.6.0-snapshot, function m/validate returns data, not boolean value true. In 0.5.1, If data is not corresponds to spec, then function m/validate returns false. In 0.6.0-snapshot, function m/validate returns in some cases value false, in some cases returns value nil (for strings for example).#2021-07-2107:21ikitommioh, that’s not good @mike1452. These might be related: • https://github.com/metosin/malli/commit/ae12531aede1ad936af7d7bde80543572cc907aehttps://github.com/metosin/malli/pull/479 … could you retest with the new SNAPSHOT (`metosin/malli-0.6.0-20210721.071739-2`) and if the problem persists, please write an issue.#2021-07-2108:26mike_ananev@ikitommi thank you for quick reply! I'll check my tests and return soon. You can use malli spec to spec your functions to catch errors, while you are developing malli.spec. 😂#2021-07-2110:11mike_ananev@ikitommi I found new strange behaviour during this cycle: edn1 -> encode-value -> json -> decode-value -> edn2 In v 0.5.1, edn1 = edn2. All uuids and zoned-date-time encoded/decoded correctly, they are data of the same type. In v 0.6.0-20210721.071739-2, edn1 != edn2 broken data for uuid and zoned-date-time - they are strings after decoding. Can't reproduce this error on a simple schema. I've more complicated schemas, but cannot publish them here. But in my code I do something like that.#2021-07-2110:12mike_ananev
(do (def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
    (def my-spec1 [:map
                   [:a :zoned-date-time]
                   [:b :string]
                   [:c :uuid]
                   [:d :int]])
    (def my-spec [:map [:s1 my-spec1] [:s2 my-spec1]])
    (def edn-value (mg/generate my-spec))
    (def encoded-value (m/encode my-spec edn-value custom-json-transformer))
    (def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
    (def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
    (def edn-value2 (m/decode my-spec encoded-value2 custom-json-transformer))
    (is (m/validate my-spec edn-value2)))
#2021-07-2112:26Ben SlessIs this example sufficient for reproduction or is it just illustrative?#2021-07-2113:03Ben SlessSome schemas are missing from the example, too#2021-07-2113:27mike_ananev@UK0810AQ2 it is illustrative#2021-07-2116:36Ben SlessI MANAGED TO RECREATE IT
(do
  (def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
  (def my-spec1 [:map
                 [:a/c1  {:optional true} :uuid]
                 [:a/c2  :uuid]
                 [:a/c3  :uuid]
                 [:a/c4  :uuid]
                 [:a/c5  :uuid]
                 [:a/c6  :uuid]
                 [:a/c7  :uuid]
                 [:a/c8  :uuid]
                 [:a/c9  :uuid]
                 [:a/c10 :uuid]
                 [:a/c11 :uuid]
                 [:a/c12 :uuid]
                 [:a/c13 :uuid]
                 [:a/c14 :uuid]
                 [:a/c15 :uuid]
                 [:a/c16 :uuid]
                 [:a/c17 :uuid]
                 [:a/c18 :uuid]
                 [:a/c19 :uuid]
                 [:a/c20 :uuid]
                 [:a/c21 :uuid]
                 [:a/c22 :uuid]
                 [:d :int]])
  (def my-spec [:map [:x/y [:map [:b/foo [:map  [:a/s1 my-spec1]]]]]])
  (def edn-value (mg/generate my-spec))
  (def encoded-value (m/encode my-spec edn-value custom-json-transformer))
  (def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
  (def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
  (def edn-value2 (m/decode my-spec encoded-value2 custom-json-transformer))
  (m/validate my-spec edn-value2))
#2021-07-2116:47mike_ananev@UK0810AQ2 yeah! This code reproduces the bug. @ikitommi do you have any suggestions?#2021-07-2116:49Ben SlessNow I can debug and investigate#2021-07-2116:49mike_ananevThe bug is when we have optional attribute in a map then process edn -> encode -> json -> decode ->edn2 is broken, edn != edn2 . This behavior is in version 0.6.0-20210721.071739-2. In 0.5.1 version all is working as expected.#2021-07-2117:42Ben SlessManaged to narrow the example down a bit:
(do
  (def custom-json-transformer (mt/transformer mt/string-transformer mt/json-transformer))
  (def my-spec1 [:map
                 [:a/c1  {:optional true} :uuid]
                 [:a/c2  :uuid]])
  (def s (m/schema my-spec1))
  (def my-spec (m/schema [:map [:x s]]))
  (def g (mg/generator my-spec))
  (def edn-value (mg/generate g {:seed 9999}))
  (def e (m/encoder my-spec custom-json-transformer))
  (def encoded-value (e edn-value))
  (def json-string-value (json/write-value-as-string encoded-value json/keyword-keys-object-mapper))
  (def encoded-value2 (json/read-value json-string-value json/keyword-keys-object-mapper))
  (def d (m/decoder my-spec custom-json-transformer))
  (def edn-value2 (d encoded-value2))
  (m/validate my-spec edn-value2))
#2021-07-2117:42Ben SlessThis returns false consistently#2021-07-2118:26Ben SlessI managed to find the commit which caused the change: 9dc8da7a0649ed93554c1fe16ea48c2bebd724ad fast collection transformers#2021-07-2118:32Ben SlessI am not sure where the bug is, but changing the implementation of -map-transformer to
(reduce
      -comp
      (map
       (fn [[k t]]
         (fn [x]
           (if-let [e (.entryAt x k)]
             (.assoc x k (t (.val e)))
             x)))
       ts))
Fixes it
#2021-07-2118:32mike_ananev👍#2021-07-2207:51Ben Slesshttps://github.com/metosin/malli/issues/480#2021-07-2208:54ikitommithanks @UK0810AQ2. I hate it when there is test to cover this. My bad. The minimal repro:
(deftest regression-480-test
  (let [value {:b #uuid"f5a54a8f-7d78-4495-9138-e810885d1cdb"}
        schema [:map [:a :int] [:b :uuid]]]
    (is (= value
           (as-> value $
                 (m/encode schema $ mt/string-transformer)
                 (m/decode schema $ mt/string-transformer))))))
#2021-07-2209:00ikitommiupdated clojars;
➜  ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.6.0-SNAPSHOT"}}}'
Downloading: metosin/malli/0.6.0-SNAPSHOT/malli-0.6.0-20210722.085801-3.pom from clojars
#2021-07-2209:03ikitommibare minmial repro btw was:
(m/decode
  [:map [:a :int] [:b :int]]
  {:b "1"}
  mt/string-transformer)
; => {:b "1"}
#2021-07-2210:37mike_ananev@ikitommi @UK0810AQ2 thank you! Now, all test are passed. We will try to take version 0.6.0-20210722.085801-3 in our production.#2021-07-2015:27mike_ananevThis new behaviour of m/validate breaks some tests in my code.#2021-07-2015:28mike_ananevWhat behaviour is expected in 0.6.0 release? Should we adapt our tests for this new behaviour of m/validate ?#2021-07-2016:01gregI've just found out that mu/update-in can take a path that includes also catn options. It's so easy and concise to alter sequential schema with Malli! :grinning_face_with_star_eyes:#2021-07-2106:44Oliver GeorgeHaving my first look at Malli today. Is there a s/assert equivalent? This is a CLJS project. I'd like nice errors while developing and elided asserts in production.#2021-07-2107:15ikitommi@olivergeorge no, there is not. would it make sense to make the instrumentation work with cljs? it’s just clj ATM. If someone has mad skills at cljs-interop, clj-kondo emitting and malli.dev could be ported to cljs.#2021-07-2109:33Oliver GeorgeIt's all new to me but I'll keep it in mind as I get more familiar with Malli.#2021-07-2109:36Oliver GeorgeAdding a s/assert macro might be close to a copy/paste of cljs.spec.alpha/assert code. If it's considered useful/desirable.#2021-07-2109:54alpoxI was looking for an s/assert equivalent as well - me in Clojure though, not CLJS#2021-07-2109:56alpoxSo from the little hint to malli.dev i gather there is some similar functionality in there? I guess I'll have to look at it a bit closer#2021-07-2212:22David LevinovHey guys, very quick and simple question, I want to combine two schemas to validate for case a and case b, and I’d rather not involve regexes, and have one of the validation schemas to have the values not be of a certain set. How would I rewrite this in a correct manner?
(def NotABC
  (m/schema
    [:map
     [:field [:not [:enum "a" "b" "c"]]]]))
Thanks!
#2021-07-2215:17ikitommilooks legit to me.#2021-07-2215:53David Levinov@ikitommi that’s exactly what I thought as well, but in the REPL:
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :not}
thanks for the response 👌
#2021-07-2215:59ikitommioh, what. Try updating the malli dependency.#2021-07-2216:13David Levinovwelp, that does indeed solve the issue, thanks haha#2021-07-2214:11Lucy Wang@ikitommi Looks like the new function instrumentation only works for clj, but not for cljs yet. E.g. mi/instrument! and dev/start would throw when I use them in cljs. Is there any plan to add cljs support?#2021-07-2214:46ikitommi@wxitb2017 help most welcome on cljs-support.#2021-07-2215:14ikitommihttps://github.com/metosin/malli/issues/482#2021-07-2302:16Lucy WangCool!#2021-07-2219:14Ben SlessInitial attempt at writing a malli based parser for EDN datalog syntax https://gist.github.com/bsless/632b4040a2b2ad7469369f52cd610c06 Can't figure out why ::clause breaks and it's getting late here. Feel free to poke at the code, suggest improvements, feedback, etc.#2021-07-2314:53Ben SlessDirect translation to spec works for parsing the forms, insights as to why it didn't work with malli most welcome, updated the gist#2021-07-2314:53Ben SlessDirect translation to spec works for parsing the forms, insights as to why it didn't work with malli most welcome, updated the gist#2021-07-2313:18robert-stuttaforddoes #malli work with #babashka ? (just did a demo for my team and our infra human is curious)#2021-07-2317:09borkdudemalli currently doesn't work (from source) in babashka#2021-07-2317:09borkdudewe have an issue here: https://github.com/babashka/babashka/discussions/906 to consider including it#2021-07-2317:26borkdude@U0509NKGK there are alternatives which currently work, such as spartan.spec (a spec drop-in replacement) and minimallist#2021-07-2317:27borkdudeif you could tell me how exactly malli would be useful for infra, please discuss in the issue, it could be an argument to include it#2021-07-2915:55robert-stuttafordtruly just pure curiosity at this point 🙂#2021-07-2314:58Ben SlessIs there a reason :ref schemas just die for regex parsing? Why do they have to be surrounded by :maybe?#2021-07-2315:20ikitommi@ben.sless refs directly in sequence schemas are disallowed, rationale: https://github.com/metosin/malli/blob/master/src/malli/impl/regex.cljc#L1-L34#2021-07-2315:21ikitommiyou can wrap anything in :schema to push it outside of the sequence.#2021-07-2316:19Ben SlessIt would be good to add in the readme that recursive seqexp reference schemas have to have another level of indirection via either maybe or schema#2021-07-2411:36ikitommidoc PRs always welcome#2021-07-2315:22ikitommiHere's an example with hiccup, which is recursive:
(def Hiccup
  [:schema {:registry {"hiccup" [:orn
                                 [:node [:catn
                                         [:name keyword?]
                                         [:props [:? [:map-of keyword? any?]]]
                                         [:children [:* [:schema [:ref "hiccup"]]]]]]
                                 [:primitive [:orn
                                              [:nil nil?]
                                              [:boolean boolean?]
                                              [:number number?]
                                              [:text string?]]]]}}
   "hiccup"])

(def parse-hiccup (m/parser Hiccup))
#2021-07-2315:33Ben SlessI'll try. I got the parser working besides ref which I just hacked around for now#2021-07-2316:15Ben SlessIt works!#2021-07-2316:16Ben SlessUpdated the gist with working version, next step is a dynamic interpreter, followed by a compiler. It's pretty cool I can use the library to define new syntax. It extends itself#2021-07-2422:57nivekuilis there a way to work with bigints in cljs? trying to coerce a string (reitit route) to one#2021-07-2504:05nivekuilbigints are a headache with transit, just ended up using Long
(defn -string->bigint [x]
  (if (string? x)
    (try
      #?(:clj  (Long/parseLong x)
         :cljs (let [x' (js/parseInt x)]
                 (if (> x' js/Number.MAX_SAFE_INTEGER)
                   (goog.math.Long/fromString x 10)
                   x')))
      (catch #?(:clj Exception, :cljs js/Error) _ x))
    x))

(defn string-transformer []
  (mt/transformer
   {:name     :string
    :decoders (assoc (mt/-string-decoders)
                     :i64 -string->bigint)
    :encoders (assoc (mt/-string-encoders)
                     :i64 mt/-any->string)}))
#2021-07-2510:28Ben SlessIn providing a custom registry to a schema schema, what gets compiled? when?#2021-07-2913:37ikitommiugh. can’t recall. you need to follow the code.#2021-07-2622:28greg@ikitommi do you expect introducing :andn? In my schema some of the properties have relations that could be verified. I do it by adding a top level :and and then modify/navigate it using indices (0 for the schema, 1... for predicates). Not a bit deal to use indices, still, the case looks similar to catn .#2021-07-2913:36ikitommisure, PR welcome.#2021-07-2708:53Ben SlessI think this level of understanding the JIT is slightly above my pay grade. If someone here has expertise on the subject it would be most welcome https://github.com/metosin/malli/pull/485#2021-07-2718:00gregI've added a tip demonstrating that transformer, the one that calculates default value based on given fn in :default-fn PR: https://github.com/metosin/malli/pull/487#2021-07-2801:30greg
(def Schema (m/schema [:map
                        [:y {:limit 20} int?]]))
Given the above schema schema: 1. How to access properties of :y? I tried 2. How to update :x to double it, e.g by passing #(* % 2)? m/update combined with m/update-properties doesn't work for the same reason as above. 3. How to convert [:y {:limit 20} int?] to [:y {:limit 20} [:and int? [:> 0]] without loosing properties.
(mu/update Schema :y (fn [s] [:and s [:> 0]])) ;; => [:map [:y [:and int? [:> 0]]]]
                                      ; I expected:  [:map [:y {:limit 20} [:and int? [:> 0]]]]
Calling the update removes the original properties. Some workaround I think would be writing custom get & update based on m/children. Doesn't look a difficult taks, just thought about asking before reinventing a wheel 🙂
#2021-07-2808:48TuomasI had a similar issue and went with an approach like
(def Schema (m/schema [:map
                       [:y [int? {:limit 20}]]]))
(m/validate Schema {:y "1"})
(m/validate Schema {:y 1})
(m/properties (mu/get Schema :y))
But I'm a novice and not really sure how this should be done
#2021-07-2810:13gregI think it would work for some cases, but it might be not a good idea to store child-owned props at schema level. Let's say I got this Schema
(def Schema (m/schema [:map
                       [:x number?]
                       [:y [number? {:default 0}]])
If I do the simple :y update to ensure its >= 0 I will get this:
(-> Schema
    (mu/update :y (fn [s] [:and s [:>= 0]])))
; => => [:map 
         [:x number?] 
         [:y [:and [number? {:default 0}] [:>= 0]]]]
And I think this kind of properties, to work properly, they need to be kept in top level of schema for given child, e.g.:
[:map 
 [:x number?] 
 [:y [:and {:default 0} number? [:>= 0]]]
So to make this work you need both: • moving props from child (`[:y]`) to schema (`[number?]`) • adding custom update anyway (to move properties from original schema (`[number?]`) to a new enclosing schema (`[:and]`) But when doing this properties maneuver, how do you know which are owned by child and which by schema? You don't unless you explicitly specify it e.g. in update method. Conceptually I think it makes more sense to keep child-owned properties at child level, and schema-owned properties at schema level.
#2021-07-2810:27gregActually I went this path, and I implemented this custom update for properties manouver when updating a schema (e.g. moving from [number?] to [:and] ), but as I wrote, I don't think it is good idea. As I think now, it really makes more sense to just add a few new utility methods: get-child, get-child-props, update-child, update-child-props, update-child-schema. It is a bit weird that child properties disappear on child's schema update. It might be a bug. Who is John Galt? :man-shrugging:#2021-07-2811:33gregI wrote a few utils fns to sort out my problems with manipulating map schemas:
(defn get-child [s k] (->> (m/children s) (filter #(= k (first %))) (first)))
(defn get-child-props [s k] (-> (get-child s k) second))
(defn get-child-schema [s k] (-> (get-child s k) (nth 2)))

(defn update-child [s key f & args]
  (let [new-children (->> (m/children s)
                          (map (fn [child]
                                 (let [[k _p _v] child]
                                   (if (= key k) (apply f child args) child)))))]
    (m/into-schema (m/type s) (m/properties s) new-children)))

(defn update-child-props [s k f & args]
  (update-child s k (fn [c]
                      (let [[k p v] c]
                        [k (apply f p args) v]))))

(defn update-child-schema [s k f & args]
  (update-child s k (fn [c]
                      (let [[k p v] c]
                        [k p (apply f v args)]))))

(comment
 (def Schema (m/schema [:map [:y {:default 20} int?]]))
 (get-child-schema Schema :y)                             ;; => int?
 (get-child-props Schema :y)                              ;; => {:default 20}
 ;; updates of child props
 (update-child-props Schema :y update :default dec)       ;; => [:map [:y {:default 19} int?]]
 (update-child-props Schema :y assoc :limit 50)           ;; => [:map [:y {:default 20, :limit 50} int?]]
 ;; update of child schema preserving child props
 (update-child-schema Schema :y (fn [s] [:and s [:> 0]])) ;; => [:map [:y {:default 20} [:and int? [:> 0]]]]
 ;; update of child schema removing the props
 (mu/update Schema :y (fn [s] [:and s [:> 0]])))          ;; => [:map [:y [:and int? [:> 0]]]]
what do you think?
#2021-07-2913:31ikitommiLooks good. Few comments: • mu/find gets the entry:
(def Schema (m/schema [:map [:y {:limit 20} int?]]))

(mu/find Schema :y)
; => [:y {:limit 20} int?]
• when calling m/into-schema you should always use m/parent instead of m/type as first argument - m/type makes a registry lookup, m/parent points directly to the IntoSchema instance that created the schema. Faster AND adds support for non-registered schemas, e.g.
(def Over6
  (m/-simple-schema
    {:type :user/over6 ;; dummy-type, not registered
     :pred #(and (int? %) (> % 6))
     :type-properties {:error/message "should be over 6"
                       :decode/string mt/-string->long
                       :json-schema/type "integer"
                       :json-schema/format "int64"
                       :json-schema/minimum 6
                       :gen/gen generate-over6}}))

(= Over6 (m/parent [Over6 {:json-schema/example 42}]))
; => true
• for now, we could have a mu/-update-entry-properties helper, PR welcome • for future (1.0.0), the entry design should be revisited, not happy how it is now
#2021-07-2914:31gregthanks a lot for the comments, I'll prepare a PR when I find spare time#2021-07-2810:37Ben SlessFigured out today a decoder can be used to reject wrong values: {:compile (fn [schema _] (let [s (set (m/children schema))] (fn [v] (s v))))}#2021-07-2913:36ikitommibit like mt/strip-extra-keys-transformer 😉#2021-07-3008:54Ben SlessSimilar, but just "forces" invalid enum values to be nil.#2021-07-2810:37Ben SlessSort of coercion#2021-07-2811:33gregI wrote a few utils fns to sort out my problems with manipulating map schemas:
(defn get-child [s k] (->> (m/children s) (filter #(= k (first %))) (first)))
(defn get-child-props [s k] (-> (get-child s k) second))
(defn get-child-schema [s k] (-> (get-child s k) (nth 2)))

(defn update-child [s key f & args]
  (let [new-children (->> (m/children s)
                          (map (fn [child]
                                 (let [[k _p _v] child]
                                   (if (= key k) (apply f child args) child)))))]
    (m/into-schema (m/type s) (m/properties s) new-children)))

(defn update-child-props [s k f & args]
  (update-child s k (fn [c]
                      (let [[k p v] c]
                        [k (apply f p args) v]))))

(defn update-child-schema [s k f & args]
  (update-child s k (fn [c]
                      (let [[k p v] c]
                        [k p (apply f v args)]))))

(comment
 (def Schema (m/schema [:map [:y {:default 20} int?]]))
 (get-child-schema Schema :y)                             ;; => int?
 (get-child-props Schema :y)                              ;; => {:default 20}
 ;; updates of child props
 (update-child-props Schema :y update :default dec)       ;; => [:map [:y {:default 19} int?]]
 (update-child-props Schema :y assoc :limit 50)           ;; => [:map [:y {:default 20, :limit 50} int?]]
 ;; update of child schema preserving child props
 (update-child-schema Schema :y (fn [s] [:and s [:> 0]])) ;; => [:map [:y {:default 20} [:and int? [:> 0]]]]
 ;; update of child schema removing the props
 (mu/update Schema :y (fn [s] [:and s [:> 0]])))          ;; => [:map [:y [:and int? [:> 0]]]]
what do you think?
#2021-07-2913:31ikitommiLooks good. Few comments: • mu/find gets the entry:
(def Schema (m/schema [:map [:y {:limit 20} int?]]))

(mu/find Schema :y)
; => [:y {:limit 20} int?]
• when calling m/into-schema you should always use m/parent instead of m/type as first argument - m/type makes a registry lookup, m/parent points directly to the IntoSchema instance that created the schema. Faster AND adds support for non-registered schemas, e.g.
(def Over6
  (m/-simple-schema
    {:type :user/over6 ;; dummy-type, not registered
     :pred #(and (int? %) (> % 6))
     :type-properties {:error/message "should be over 6"
                       :decode/string mt/-string->long
                       :json-schema/type "integer"
                       :json-schema/format "int64"
                       :json-schema/minimum 6
                       :gen/gen generate-over6}}))

(= Over6 (m/parent [Over6 {:json-schema/example 42}]))
; => true
• for now, we could have a mu/-update-entry-properties helper, PR welcome • for future (1.0.0), the entry design should be revisited, not happy how it is now
#2021-07-2909:43danielnealis there an malli.util/assoc that will allow to set the properties and schema of a key#2021-07-2909:43danielnealor a different function?#2021-07-2910:04gregThat was part my question I posted https://clojurians.slack.com/archives/CLDK6MFMK/p1627435833075200. I did not found a way of doing it at Malli. But it is fairly easy to do using m/children. If you have a look at my other https://clojurians.slack.com/archives/CLDK6MFMK/p1627471982078700?thread_ts=1627435833.075200&amp;cid=CLDK6MFMK on this channel, I wrote a few utility methods. One of them is update-child-props. It was a first draft, I modified them since then, but even with it, you can do what you want, e.g.
(update-child-props (m/schema [:map [:y {:default 20} int?]]) :y update :default dec) 
;; => [:map [:y {:default 19} int?]]

(update-child-props (m/schema [:map [:y {:default 20} int?]]) :y assoc :limit 50)           
;; => [:map [:y {:default 20, :limit 50} int?]]
#2021-07-2912:57gregUnless I misunderstood "allow to set the properties and schema of a key" 😅#2021-07-2913:34ikitommimaybe:
(mu/assoc 
  [:map [:y {:old true} int?]] 
  [:y {:new true}] string?)
; => [:map [:y {:new true} string?]]
#2021-07-2914:28danielnealaha, thanks!#2021-07-2913:43ItayHi, I posted this in the #reitit channel but still did not get any response. No reply on the issue I opened either. Since it is Malli related, I thought perhaps I could get some feedback here. Thanks.#2021-07-2913:44ikitommi@itaysabato I can take a look at that today/tomorrow#2021-08-0208:22ItayGood morning 🙂 Any news regarding this issue?#2021-08-0213:02ikitommitotally forgot, sorry. will try to look later today.#2021-08-0306:01ikitommidid a PR, might fix it. Depending on the exceptations#2021-08-0308:17ItayThank you for this! Looks like it did. I commented on one line that I didn't get, but the rest looks solid to me.#2021-08-0309:27ikitommianswered. I'm not near computer, if you could verify the old behavior (does it strip extras or not), would help. Thanks.#2021-08-0810:34ItaySorry, I was away for a few days. Is it still relevant to test the old strip keys behavior? I will soon try the fixed version#2021-07-2913:45Itay@ikitommi That would be lovely, thank you!#2021-07-2913:55ikitommiquick poke on instrumentation: malli.dev/start! will now collect function schemas from all public vars from all loaded namespaces.#2021-07-2913:56ikitommiI would like to add Var-watches too, so any Var change would re-trigger instrumentation of that Var. But didn’t work in the 30min timebox I had for it.#2021-07-2913:57ikitommianyway, dx should be better than before. feedback welcome on the instrumention. If anyone has tried that…#2021-07-2913:57ikitommi#2021-07-2914:02ikitommigoal is to do the first version of pretty error reporting for 0.6.0, to celebrate the two year birthday of the initial demo of that (https://github.com/metosin/malli/issues/19). Making open source 1h here and there isn’t the fastest way to finish stuff 😎 #2021-07-2914:02Noah Bogarthey all, just started using malli in a new part of my project, and i’m struggling to check if a value is a function#2021-07-2914:03Noah Bogart
(def Step
  [:map {:closed true}
   [:uuid uuid?]
   [:continue-fn fn?]
   [:complete? boolean?]
   [:phase keyword?]])
#2021-07-2914:03ikitommishould work#2021-07-2914:03ikitommifn? was added few days ago thou, it was missing. you should depend on the latest sha#2021-07-2914:04Noah Bogartah that’s probably it, i’m using the clojars version#2021-07-2914:04ikitommi[:fn fn?] works also. :fn is the escape hatch, “any function”#2021-07-2914:07Noah Bogartthat works! thanks so much#2021-07-2914:16ikitommiimmutable clojars-snapshot of the latest stuff:
➜  ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.6.0-SNAPSHOT"}}}'
Downloading: metosin/malli/0.6.0-SNAPSHOT/malli-0.6.0-20210729.141459-4.pom from clojars
#2021-07-2914:17ikitommibtw, @nbtheduke there is a new guide for function schmemas at https://github.com/metosin/malli/blob/master/docs/function-schemas.md#2021-07-2914:18Noah Bogartmissed that there were further docs, i’ll read over that now#2021-07-3010:05Ben SlessCurious why change from ArrayList to LinkedList in the transformers' implementation. Aren't arrays better for iteration and cache?#2021-07-3010:26ikitommiLinkedList has much simpler iterator impl. Didn't see any visible perf difference thou.#2021-07-3011:05Ben SlessThis is slightly theoretical, but ArrayLists have a smaller memory footprint and their backing array is contiguous in memory which ought to give better performance. Also, although the iterator implementation is simpler, ArrayLists would probably benefit from branch predictions, while pointer-chasing with linked lists gives you no guarantees#2021-07-3011:06Ben Slesshttps://stackoverflow.com/questions/8436489/using-a-linkedlist-or-arraylist-for-iteration#2021-07-3011:07Ben SlessModern CPUs like arrays 😞#2021-07-3015:36gregI guess you talk about https://github.com/metosin/malli/commit/dd19b9e. I, same as @UK0810AQ2, would choose ArrayList, especially that we know the size of data. Theoretically it should be faster in allocation of memory and further iteration, but well, always good to see relevant data. Personally I don't care about performance that much so I don't mind either of them. Still, I'm curious, @U055NJ5CC why the iterator's impl matters to you since its interface is the same?#2021-07-3016:04ikitommiCollection transformers will be rewritten by @UK0810AQ2 to remove all iteration. But, perf comes from impl, not perf. Was really iteresting read, I still think the iteration is faster on LL, but, irrelevant here. As a generic data structure AL is much better.#2021-07-3017:51Ben SlessI don't know if the composition based solution will be faster. Will try to poke at it with jmh#2021-07-3111:39Ben SlessRight, jmh is running, results in a few hours#2021-07-3113:56Ben Sless
([:map-transformer/linked-list {:count 1} [6.156812627775496E7 "ops/s"]]
 [:map-transformer/linked-list {:count 2} [3.2994990990360465E7 "ops/s"]]
 [:map-transformer/linked-list {:count 4} [1.7037915234624855E7 "ops/s"]]
 [:map-transformer/linked-list {:count 8} [7557295.433684901 "ops/s"]]
 [:map-transformer/linked-list {:count 16} [1566793.27656629 "ops/s"]]
 [:map-transformer/linked-list {:count 32} [704492.6211734915 "ops/s"]]

 [:map-transformer/array-list {:count 1} [6.3440679176932156E7 "ops/s"]]
 [:map-transformer/array-list {:count 2} [3.439043251403162E7 "ops/s"]]
 [:map-transformer/array-list {:count 4} [1.689684037729955E7 "ops/s"]]
 [:map-transformer/array-list {:count 8} [7481296.308118063 "ops/s"]]
 [:map-transformer/array-list {:count 16} [1550320.2392137954 "ops/s"]]
 [:map-transformer/array-list {:count 32} [713659.2962506406 "ops/s"]]

 [:map-transformer/comp {:count 1} [8.510126517455931E7 "ops/s"]]
 [:map-transformer/comp {:count 2} [4.2607291886022285E7 "ops/s"]]
 [:map-transformer/comp {:count 4} [1.9971178945141207E7 "ops/s"]]
 [:map-transformer/comp {:count 8} [8451028.653528532 "ops/s"]]
 [:map-transformer/comp {:count 16} [1599471.9346010373 "ops/s"]]
 [:map-transformer/comp {:count 32} [737048.961960025 "ops/s"]])
#2021-07-3114:41Ben SlessAnother avenue I want to explore is creating from an array. Can't avoid an array copy anyway, so why not copy just once?#2021-07-3119:24Ben Slessmore plausible to achieve with a tuple transformer, don't know if can be done with map#2021-08-0105:32Ben SlessI think I found a faster way to transform a tuple:
(defn -tuple-transformer [ts]
  (let [ts (map (fn [[k t]] (let [i (int k)] (fn [^objects arr] (aset arr i (t (aget arr i))) arr))) ts)
        t (apply -comp ts)]
    (fn [^clojure.lang.APersistentVector x]
      (clojure.lang.LazilyPersistentVector/createOwning (t (.toArray x))))))
Will check some arities combinations, initial tests look promising
#2021-07-3011:25danielnealIf you want to have different error messages for different predicates in a malli schema, would it be wrong to do this#2021-07-3011:25danielneal
[:and [:fn {:error/message "error 1"} some-constraint]
[:fn {:error/message "error 2"} some-other-constraint]]
#2021-07-3013:56danielnealwhat’s the schema for a malli schema? e.g. for a key in a map there’s a key in a map that can be a schema [:map [:a int?] [:spec :???schema???]]#2021-07-3014:35respatializedThe term I've used to describe this is alternately "meta-schema" or "self-describing schema" - there isn't one yet for malli schemas.#2021-07-3014:43danielnealah ok, thanks#2021-07-3014:44danielnealI’ll just use any? I guess#2021-07-3013:56danielnealcouldn’t figure out what to search for#2021-07-3013:57danielnealthe word schema appears a lot 😉#2021-07-3014:03danielnealis it just malli/schema??#2021-07-3014:04danielnealhm no that doesn’t seem to work#2021-07-3014:13Noah Bogartdoes malli care about stuff like volatile!? I have a schema for a map and i’m thinking of changing it to a volatile, so how do I put that in my schema? I changed the schema to [:volatile [:map …]] but that didn’t work#2021-07-3014:35respatializedThe term I've used to describe this is alternately "meta-schema" or "self-describing schema" - there isn't one yet for malli schemas.#2021-08-0108:31Ben Slesshot take: :fn schema should be deprecated#2021-08-0108:49ikitommiwhy is that?#2021-08-0109:12Ben Sless95% of the instances I saw it used the programmer should have used properties or defined a -simple-schema#2021-08-0109:14Ben SlessIt is used in a way too general manner, usually one which impairs serialization and generation#2021-08-0109:15Ben SlessWe can write guides and improve documentation and give workshops and do backflips and pay for ads on the superbowl, people will still use :fn because it's easy#2021-08-0109:20Ben Slessgenerators and transformers suffer from this, big time#2021-08-0109:21Ben SlessAnd it makes the code opaque#2021-08-0110:56Ben SlessI did say this was a hot take 🙃#2021-08-0111:20gregI think -simple-schema is not always what you want to do. And yes :fn is pretty handy, especially if you don't need transformers and generators but only validation. I'm not sure it would be a good move to force people to use -simple-schema just for validation (e.g. involving two or more map entries).#2021-08-0111:40Ben SlessThis is absolutely me speaking from frustration, not a definitive or proscriptive solution#2021-08-0111:45Ben SlessThe problem isn't using fn but using it where [:string {:max 10 }] would do
#2021-08-0111:50Ben SlessI also see predicate functions used a lot where decoders would be better#2021-08-0114:56gregOk, make total sense now. I just took it as a proposal of change. I need to develop a sense of humour I guess 😅#2021-08-0115:08Ben SlessSee the "hot take" prefix#2021-08-0115:37gregI missed that :face_palm:😅#2021-08-0115:10Ben SlessFrustrated by half a day fighting with badly written and used malli schemas Everything can and will be abused#2021-08-0115:41gregI don't know you particular case, so just guessing... Maybe organising some internal training showing how to use Malli could avoid that kind of tool misuse in the future 😉🚀#2021-08-0116:28Ben SlessI am actually working on internal training which I hope I'll be able to release mostly as a contributed getting started guide#2021-08-0115:13Noah BogartI suspect this also has to do with docs. It can be hard to fully enumerate the possibilities of a given API, and something like :fn just works. #2021-08-0115:36gregI agree with this point. @UK0810AQ2 tbh with you, I understand the points that frustrates you, but I wouldn't be two weeks ago. The thing is that Malli is nice to use once you know it, but it is fairly confusing when you start. I think that it might be the source of the problem, that people are using the simplest solutions they know, because they simply don't know all the features of the tool. Let's take my example. I used spec before, but haven't used spec.tools nor prismatic schema. Some of the concepts were simply new to me. E.g. transformers, there are at least four ways of transforming values with Malli, and it is not obvious what they are and how they work. I discovered it by reading the docs, examples, Malli impl and reading the Tommis answers here. I don't have daily job and I could invest a bit of time, still not everyone can afford that much of investment, so most of the time, I guess, people pragmatically makes things done 🙂 In this particular topic, using Malli transformers, I intend to make a comprehensive writing at some point to make the process easier for others. I think in case of :fn and simple-schema case is the same, people might not be aware when/where to use what 🙂#2021-08-0116:30Ben SlessI agree. Malli's docs are comprehensive but like man pages they don't tell you where to start, they don't guide you towards "I want to do X"#2021-08-0116:14zxspectrris there a more idomatic way of defining a map that must have a key but the value can be nullable. Here's what I've currently got:
(defn ? [predicate] [:or predicate nil?])

(def MyThing
     [:map      
      [:someValue (? string?)]])
#2021-08-0116:15zxspectrrOnly reason I want to force the existence of a key so it appears in a JSON response with null as it's value#2021-08-0116:51alpoxI'm not 100% sure but would that not be [:someValue [:maybe string?]]#2021-08-0117:04zxspectrroh, I didn't realise there's a [:maybe]#2021-08-0117:04zxspectrrnice#2021-08-0117:04zxspectrryep that works, ty 🙂#2021-08-0117:05zxspectrrmy experience with malli so far has been quite positive#2021-08-0213:05ikitommiwip#2021-08-0218:54Ben SlessIs it integrated with kondo as well?#2021-08-0219:51ikitommimalli.dev is, the red underline & bump are from kondo. The pretty printer is currently just for runtime errors.#2021-08-0213:06ikitommithe error formatters use fipp, look like this:
(defmethod v/-format ::m/invalid-input [_ _ {:keys [args input]} printer]
  {:body
   [:group
    (-block "Invalid function arguments:" (v/-visit args printer) printer) :break :break
    (-block "Input Schema:" (v/-visit input printer) printer) :break :break
    (-block "Errors:" (-errors input args printer) printer) :break :break
    (-block "More information:" (v/-color :link "" printer) printer)]})
#2021-08-0213:08ikitommithere will be pretty/reporter to just prints the errors and pretty/thrower to throw ’em.#2021-08-0213:13Noah Bogartthat looks great#2021-08-0213:13Noah Bogartis there a better idiom for this?
(defn Step? [step]
  (if (m/validate Step step)
    step
    (throw (Exception.
             (->> step
                  (m/explain Step)
                  (me/humanize))))))
#2021-08-0213:19danielnealhaha I literally wrote that exact function 5 mins ago and was wondering the same thing#2021-08-0213:57ikitommi• you could create a validator & explainer before, if perf matters • I recommend throwing ex-imfo with :type , easier to catalog the errors • humanize can fail in some corner cases, should be robust, fixes welcome • there could be a helper for this in the future in malli, using m/-fail! which works well with the upcoming pretty printer#2021-08-0214:02ikitommiMalli internal errors just throw the minimum set of data (to be fast), to be post-processed when rendering the error. e.g. explain and humanize are pure, can be done on rendering, if the value & schema are present at ex-data. Might not be relevant on actual projects.#2021-08-0214:04ikitommiI haven't heard of a robust way to override the repl default exception handler (from the repl) - would allow pretty errors for all Clojure: just throw data and add custom renderers for all :types & exception classes#2021-08-0214:06Noah Bogartso something like this?
(defn SimpleStep? [step]
  (if (validate-step @step)
    step
    (throw (ex-info "Step isn't valid" {:type (explain-step @step)}))))
#2021-08-0214:07Noah Bogartgood point about the validator and explainer, thanks#2021-08-0218:58Ben SlessI'm leaning towards this form:
(defn needs-a-good-name
  [s]
  (let [s (m/schema s)
        v (m/validator s)
        d (m/decoder s ,,,) ;; optional?
        e (m/explainer s)]
    (fn [x]
      (let [x (d x)]
        (if (v x)
          x
          (throw (ex-info "invalid something" (e x))))))))
More generally, you can pass two functions, on-success and on-fail, then you can decide if you want to throw or drop or invalid data
#2021-08-0218:59Ben SlessSomething like
(defn needs-a-good-name
  [s respond raise]
  (let [s (m/schema s)
        v (m/validator s)
        d (m/decoder s ,,,) ;; optional?
        e (m/explainer s)]
    (fn [x]
      (let [x (d x)]
        (if (v x)
          (respond x)
          (raise e x))))))
#2021-08-0311:03gregI think these two examples could be posted somewhere#2021-08-0311:13Ben SlessI'll clean it up a bit and PR it to the tips? Maybe we should add a cookbook#2021-08-0219:47ikitommicoercer? did you mean (raise (e x))? Look good anyway 👍 👍 👍#2021-08-0220:35Ben Slesscoercer is probably good. Wasn't sure about passing the original value or just the report to raise#2021-08-0308:01vlaaadHi! Is there a malli equivalent of s/keys* ?#2021-08-0309:18ikitommiNot yet, I guess the could be...#2021-08-0308:07vlaaadside note:
(let [s [:catn [:rest [:? any?]]]]
  (m/unparse s (m/parse s [])))
=> [nil] ;; should be []
#2021-08-0309:18ikitommiOh, it's a bug, issue /& PR welcome!#2021-08-0308:54Ben SlessAny idea how to make the transformer tests not rely on map order? https://github.com/metosin/malli/pull/497#2021-08-0309:17ikitommi@ben.sless looks great! Commented the PR, ideas about the order & stuff#2021-08-0315:20Noah Bogarthow do i do this?
(def PhaseStepSchema
  (mu/merge
    BaseStepSchema
    [:map {:closed true}
     [:phase qualified-keyword? {:namespace :phase}]]))
#2021-08-0315:21Noah Bogarti want the :phase value to be a namespace-qualified keyword, with the namespace :phase , aka :phase/action#2021-08-0316:44Noah Bogartsolution i found: [:phase [:qualified-keyword {:namespace :phase}]]#2021-08-0317:42Ben SlessI came across an extremely weird phenomenon which I don't think should happen
1. Unhandled java.lang.VerifyError
   Stack map does not match the one at exception handler 1251 Exception
   Details: Location:
   malli/core_test$fn__30148$fn__32525.invoke()Ljava/lang/Object; @1251:
   astore_3 Reason: Type 'java/lang/Throwable' (current frame,
   locals[6]) is not assignable to 'clojure/lang/Keyword' (stack map,
   locals[6]) Current Frame: bci: @1175 flags: { } locals: {
   'malli/core_test$fn__30148$fn__32525',
   'clojure/lang/IPersistentVector', null, null, 'java/lang/Object',
   null, 'java/lang/Throwable' } stack: { 'java/lang/Throwable' }
   Stackmap Frame: bci: @1251 flags: { } locals: {
   'malli/core_test$fn__30148$fn__32525',
   'clojure/lang/IPersistentVector', 'java/lang/Object', null,
in accumulating errors #84 Will try another JVM version
#2021-08-0317:43Ben SlessThis happens in the same test, on both Java 15 and 11, only after I run it twice#2021-08-0317:44Ben SlessHappens on java 8, too#2021-08-0409:11Ben SlessI managed to narrow it down to multi-schemas#2021-08-0409:12Ben SlessSpecifically, the last bit#2021-08-0409:12Ben Sless
(let [schema [:multi {:dispatch first}
                  [:human [:cat [:= :human]]]
                  [:bear [:cat [:= :bear] [:* :int]]]
                  [::m/default [:tuple :string :string]]]]

      (testing "validate"
        (is (m/validate schema [:human]))
        (is (m/validate schema [:bear 1 2 3]))
        (is (m/validate schema ["defaultit" "toimii"]))
        (is (not (m/validate schema [:so :invalid]))))

      (testing "explain"
        (is (not (m/explain schema [:human])))
        (is (not (m/explain schema [:bear 1 2 3])))
        (is (not (m/explain schema ["defaultit" "toimii"])))
        (is (results= {:schema schema,
                       :value [:so :invalid],
                       :errors [{:path [::m/default 0], :in [0], :schema :string, :value :so}
                                {:path [::m/default 1], :in [1], :schema :string, :value :invalid}]}
                      (m/explain schema [:so :invalid]))))

      (testing "parser"
        (is (= (miu/-tagged :human [:human]) (m/parse schema [:human])))
        (is (= (miu/-tagged :bear [:bear [1 2 3]]) (m/parse schema [:bear 1 2 3])))
        (is (= (miu/-tagged ::m/default ["defaultit" "toimii"]) (m/parse schema ["defaultit" "toimii"])))
        (is (= ::m/invalid (m/parse schema [:so :invalid])))))
#2021-08-0409:12Ben SlessSomething here is very wrong#2021-08-0409:13Ben SlessEven more specifically, it's the parser#2021-08-0409:14gregI've been using REBL recently and I noticed that Malli schemas are printed as regular one line strings, which is annoying. Although I found it is possible to provide custom viewer to REBL. Example:
(defn schema? [s]
  (m/schema? (try (m/schema s) (catch Exception _))))

(defn setup-rebl-viewers []
  (require '[cognitect.rebl])
  (cognitect.rebl/update-viewers
   {:malli/schema {:pred #'schema?
                   :ctor (fn [v]
                           (-> (m/form (m/schema v))
                               (fipp.edn/pprint {:width 80})
                               (with-out-str)
                               (cognitect.rebl.renderers/string-code-viewer)))}}))
#_(setup-rebl-viewers)
This is just a simple pretty print, but we could apply other strategies for formatting. If you have an idea what's the best way of formatting Malli schemas, please let me know in the comment.
#2021-08-0409:15Ben Slessdatafy?#2021-08-0409:17Ben SlessExtend datafy to schemas and return form?#2021-08-0410:17gregHmm, I'll give it a try later :thinking_face:#2021-08-0410:17gregDo you datafy malli schemas? Could you post an example?#2021-08-0410:51Ben SlessBit of black magic:
(require '[clojure.core.protocols :as p]
         '[malli.core :as m])


(extend-protocol p/Datafiable
  Object
  (datafy [x]
    (cond
      (satisfies? m/Schema x)
      (do
        (extend-protocol p/Datafiable
          (class x)
          (datafy [y] (m/-form y)))
        (p/datafy x))

      (satisfies? m/IntoSchema x)
      (do
        (extend-protocol p/Datafiable
          (class x)
          (datafy [y] (m/-form (m/schema y))))
        (p/datafy x))

      :else x)))
#2021-08-0413:55gregI think it could be simplified to
(extend-protocol p/Datafiable
  Object
  (datafy [x]
    (cond
      (satisfies? m/Schema x) (p/datafy (m/form x))
      (satisfies? m/IntoSchema x) (p/datafy (m/form (m/schema x)))
      :else x)))
Still as I mentioned in the other post (https://clojurians.slack.com/archives/C03S1KBA2/p1628084493261600?thread_ts=1628082589.258900&amp;cid=C03S1KBA2), I've got a feeling extending Object is not a good idea 😬
#2021-08-0501:01emccueor, add some metadata to your schemas that extends the protocol#2021-08-0501:02emccuethats probably the best way#2021-08-0501:04emccue(with-meta [:map [:x int?]] ^{`p/datafy (fn [this] (p/datafy (m/form this)))})#2021-08-1022:36gregThanks for posting example. Still I think that would be yet another extreme if I add my own m/schema to keep it everywhere...#2021-08-1022:42gregActually, maybe this should be added to Malli itself, to m/schema :thinking_face: What do you think @U055NJ5CC about making Malli schemas datafy-able? (it is just an idea, I'm not sure is it good or bad, i'm just trying to figure out best practices of using these new protocols for my own and in this very thread it stroke me Malli could be a good place to define that datafy behaviour)#2021-08-0409:55danielnealI’m trying to make a recursive schema#2021-08-0409:55danielnealdoing something like this (simplified)#2021-08-0409:55danielneal
(malli/validate 
  [:vector {:registry {::property
                       [:map
                        [:property/type keyword?]
                        [:property/group {:optional true}
                         [:vector ::property]]]}}
   ::property]
  [{:property/type :string}
   {:property/type :key}])
#2021-08-0409:55danielnealbut I get a stackoverflowerror#2021-08-0409:56danielnealI’m not sure model this kind of thing correctly#2021-08-0409:57danielnealoh I see it, i need :ref ?#2021-08-0410:06delaguardoyes#2021-08-0410:51Ben SlessBit of black magic:
(require '[clojure.core.protocols :as p]
         '[malli.core :as m])


(extend-protocol p/Datafiable
  Object
  (datafy [x]
    (cond
      (satisfies? m/Schema x)
      (do
        (extend-protocol p/Datafiable
          (class x)
          (datafy [y] (m/-form y)))
        (p/datafy x))

      (satisfies? m/IntoSchema x)
      (do
        (extend-protocol p/Datafiable
          (class x)
          (datafy [y] (m/-form (m/schema y))))
        (p/datafy x))

      :else x)))
#2021-08-0517:04Ben SlessI forgot dealing with time was such a pain in the#2021-08-0615:55richiardiandreaHello there, I know this is not the right channel but I have a question about spec-tools Is there out there a library for creating graphviz svg from a spec by any chance?#2021-08-0617:06ikitommiI recall there is. Haven't used thou.#2021-08-0622:45richiardiandreaok thanks I'll google it better, could not find it#2021-08-0807:04ikitommimaybe https://github.com/jebberjeb/specviz?#2021-08-0915:36richiardiandreaAh yeah I tried that one, no dice, I thought I saw a spec-tools specific one...however Malli looks super cool and I should try it out 😄 Thanks for the link anyways!#2021-08-0620:00ikitommiHumanized errors might finally work as expected. Fixes all known issues and is much simpler implementation. If someone can still crash it with some input, I’m all 👂s. Maybe we can now remove the safe-humanize from projects 🙂 https://github.com/metosin/malli/pull/502#2021-08-0620:03ikitommibig change is that the humanized form is taken from the first failure. e.g.
(-> [:map [:x [:and [:map [:y :any]] seq?]]]
    (m/explain {:x {}})
    (me/humanize))
;{:x {:y ["missing required key"]
;     :malli/error ["should be a seq"]}}

(-> [:map [:x [:and seq? [:map [:y :any]]]]]
    (m/explain {:x {}})
    (me/humanize))
; {:x ["should be a seq"]}
#2021-08-0620:31naomarikthat's awesome, really needed that!#2021-08-0620:03ikitommisame with sequences & sets.#2021-08-0620:05Noah Bogartdoes humanize work with records now?#2021-08-0620:05Noah Bogartwell, i guess i should say that that’s part of explain, not humanize so nvm lol#2021-08-0620:06ikitommirecords, good question, will test#2021-08-0620:08ikitomminope:
actual: java.lang.UnsupportedOperationException: Can't create empty: malli.error_test.Horror
 at malli.error_test.Horror.empty (error_test.cljc:549)
    clojure.core$empty.invokeStatic (core.clj:5236)
    malli.error$_assoc_in.invokeStatic (error.cljc:147)
#2021-08-0620:08Noah Bogarta simple (into {} rec) works for me right now#2021-08-0620:08ikitommiidea how to create empty records?#2021-08-0620:09Noah Bogarti don’t believe you can, lol#2021-08-0620:09Noah Bogarthttps://clojure.atlassian.net/browse/CLJ-1975#2021-08-0620:09Noah Bogartthat’s for spec but discusses the issue#2021-08-0620:14ikitommihttps://github.com/metosin/malli/pull/502/commits/19c3bbe34d2d0831ba6d4a3138debd36f3642267#2021-08-0620:14ikitommithe humanized for doesn’t preserve the Records, but then again, it forces all sequences to vectors. it’s for… humans.#2021-08-0620:15Noah Bogarthell yeah, thank you so much#2021-08-0620:24ikitommimerged in master#2021-08-0620:39ikitommimerged also the new pretty (schema error) printer into master. Not expound, but one could build such on top of this.#2021-08-0817:02ikitommiit’s out! (actually 0.6.1 as there was a missing dependency)#2021-08-0817:55emccueGetting what is perhaps a simple error, but#2021-08-0817:56emccue
(ns dev.mccue.domain.user
  (:require
    [malli.instrument :as instrument]))

;; ----------------------------------------------------------------------------
(def User [:and [:map
                 [:user/email :string]
                 [:user/password-hash :string]]
           [:fn {:error/message "should have user metadata"}
            (fn [o] (= (type o) ::user))]])

;; ----------------------------------------------------------------------------
(defn create
  {:malli/schema [:=>
                  [:cat [:map
                         [:email :string]
                         [:password-hash :string]]]
                  User]}
  [{:keys [email password-hash]}]
  ^{:type ::user}
  {:user/email         email
   :user/password-hash password-hash})

;; ----------------------------------------------------------------------------
(def ^{:malli/schema [:=> [:cat User] :string]}
  email
  :user/email)

;; ----------------------------------------------------------------------------
(def
  ^{:malli/schema [:=> [:cat User] :string]}
  password-hash
  :user/password-hash)

;; ----------------------------------------------------------------------------
(instrument/collect!)

;; ....
(malli.dev/start! {:report (malli.dev.pretty/reporter)})
#2021-08-0905:03TuomasI could reduce it down to be about malli.dev.pretty/reporter and :fn but couldn't figure out the reason for overflow.
(defn identity-42
  {:malli/schema [:=> [:cat [:fn (fn [n] (= 42 n))]] :int]}
  [a] a)

(instrument/collect!)
(malli.dev/start! {:report (malli.dev.pretty/reporter)})
(identity-42 41) ; Execution error (StackOverflowError) at fipp.ednize/override? (ednize.clj:12)

(malli.dev/stop!)
(malli.dev/start!)
(identity-42 41) ; :malli.core/invalid-input {:input [:cat [:fn #funct ...
#2021-08-0817:56emccuethis is what i load in (requiring malli dev and malli pretty in repl)#2021-08-0817:56emccue
(password-hash {})
Execution error (StackOverflowError) at fipp.ednize/override? (ednize.clj:12).
null
#2021-08-0817:57emccueand this is what happens when i try invalid input#2021-08-0817:57emccue
(password-hash (create {:email "A" :password-hash "a"}))
=> "a"
#2021-08-0817:57emccuevalid input works fine though#2021-08-0817:57emccuejust changed to [metosin/malli "0.6.1"]#2021-08-0910:21ikitommioh, it's the function values, need to short-circuit on them. Should be easy to fix.#2021-08-0910:25ikitommithough experiment: • persist function schemas into edn/file (var->schema) • write a clj-kondo plugin/hook that looks from afile if a Var has a malli schema defined. If it has runs that validation (inputs & outputs) to it and reports. Static analysis with full malli :thinking_face: 🚀 parrot #2021-08-0910:29Ben Slessseparating type and code for functions seems like a footgun waiting to go off#2021-08-0910:29Ben Slesssomeone will change one and not the other#2021-08-0910:30borkdudeI think you could have some kind of development runtime hook that writes clj-kondo type config as you go. But if I understand correctly malli already has this#2021-08-0910:31borkdudeone issue is that statically visible values are not the runtime values, so validating on those has different behavior#2021-08-0910:31ikitommiyes, malli.dev/start! re-writes the the clj-kondo config on any change to the function registry#2021-08-0910:31ikitommicould emit the new var->malli-schema file too at the same time.#2021-08-0910:32ikitommi(and malli.dev/stop! removes the file(s))#2021-08-0910:34ikitommiyes, that works with simple/demo cases. would need typedclojure to follow the types for real? and actual runtime to track the values for real?#2021-08-0910:36ikitommiI guess one could just add new keys to the clj-kondo config too? e.g. :malli/schema [:=> [:cat :int] :int]]#2021-08-0910:37ikitommiwould be just one config file then, always up-to-date.#2021-08-0910:38borkdudeyou mean type aliases?#2021-08-0910:40ikitommi
{:linters 
	{:type-mismatch 
		{:namespaces 
			{malli.demo 
				{plus 
					{:arities 
						{1 {:args [:int]
				            :ret :int
				            :malli/schema [:=> [:cat int?] pos-int?]}}}}}}}}
#2021-08-0910:41ikitommi… for the plugin to read from.#2021-08-0910:44borkdudebut when your plugin is called with the input types, it would have to do something similar to the clj-kondo "type" system right#2021-08-0910:45borkdudeso why not convert the schema to the clj-kondo type system immediately#2021-08-0911:18ikitommiit is converted already to clj-kondo type system. In top of that, using second round of malli-vqlidation, one could catch more, like integer min&max sizes, collection limits, closed maps, multis, sequences etc. The code would require access to: 1. function arguments 2. the :malli/schema value (from linter config). #2021-08-0911:19borkdudeand you would need to invoke malli itself as well right?#2021-08-0911:19ikitomminot sure if this is anyhow useful, but might be doable? At least emitting the malli-schema to the linter config would be a +1loc in malli.#2021-08-0911:19ikitommiyes#2021-08-0911:20borkdudein that case malli would have to be built into the clj-kondo or clojure-lsp binary, unless you run it on the JVM#2021-08-0911:21borkdudewhat you could do, as it is now, is programmatically generate hooks for each var that has a malli spec#2021-08-0911:21borkdudethere you can have access to the arguments and do whatever you like, throw exceptions. The hooks already have access to the linter config#2021-08-0911:22borkdudeand then you could write some validations like malli but in user space, just as a proof of concept#2021-08-0911:22borkdudeor you can fork clj-kondo and add malli to the type system and explore until you reach some interesting conclusions#2021-08-0911:22ikitommisounds fun#2021-08-0911:23borkdudeI'm also open to clj-kondo type system improvements, there are a few low hanging fruits perhaps#2021-08-0915:02ikitommi... also, if the config-file could be used as a.simple database, the tool could run Malli's check once for each var (gen-test) and mark :malli/check with the result (nil or error) - "function does not conform to it's schema, with arguments [0 -1], the result is -1, which is not a positive integer"#2021-08-0915:24Noah Bogartis it possible to enable generative testing (`{::m/function-checker mg/function-checker}`) for all function schemas during test runs without adding a (when (= :test (:env app)) ...) to every schema manually?#2021-08-0916:35ikitommithere is an issue to allow setting default options. Before that, you could redefine the var where it's read / defined?#2021-08-0916:35ikitommidirty, but works#2021-08-0916:35Noah Bogartyeah, maybe i’ll try that#2021-08-0916:35ikitommiAnd please write an issue#2021-08-0915:39richiardiandreaQuestion, say I want to try Malli but I don't have the fire power to rewrite all the specs...What would be the best "migration strategy" there? We instrument every function in dev mode and tests and we use specs for our domain model and http param validation at the moment. What I am worried about the former and the following - how can I instrument a function that has been partially covered with spec and partially with Malli?#2021-08-0916:37ikitommicreate a spec->malli converter, PR that and enjoy the ride? :face_with_cowboy_hat:#2021-08-0916:38ikitommiNot sure if the intstrumentatons stack, could just work?#2021-08-1003:34JoelI would think ':b' could be in square brackets: [:a {:x "s"} [:b]], but it doesn't work, i'd like to be able to nest :cat/n
(-> [:cat
       [:enum :a]
       [:map [:x string?]]
       [:cat [:enum :b]]]
      (mc/explain
        [:a {:x "s"} :b])
      (me/humanize))
#2021-08-1004:44ikitommi@joel380 from the README:
As all these examples show, the "seqex" operators take any non-seqex child schema to mean a sequence of one element that matches that schema. To force that behaviour for a seqex child :schema can be used:

(m/validate
  [:cat [:= :names] [:schema [:* string?]] [:= :nums] [:schema [:* number?]]]
  [:names ["a" "b"] :nums [1 2 3]])
; => true

;; whereas
(m/validate
  [:cat [:= :names] [:* string?] [:= :nums] [:* number?]]
  [:names "a" "b" :nums 1 2 3]) 
; => true
#2021-08-1004:44ikitommie.g.
[:cat
 [:enum :a]
 [:map [:x string?]]
 [:schema [:cat [:enum :b]]]]
#2021-08-1005:45ikitommi@emccue fixed in master. Also, you don’t need the malli.instrument/collect!, malli.dev/start! calls that.#2021-08-1005:45ikitommihttps://github.com/metosin/malli/pull/509#2021-08-1012:49emccueBut it won't call it on reload if i'm using the metadata form for the schema right?#2021-08-1013:11ikitommicurrently only on start!. I tried to add var-watching to the once-registered vars, but didn't have the skills to do that. start! & stop! should play nice with reloaded repl thou.#2021-08-1012:01Darnaroth DarnarowthQuestion, is it possible to add sci options on a custom decoder, or any ability to set sci options globally?#2021-08-1012:20Karol WójcikHi! How can I check if x satisfies the protocol y in malli metadata schema?#2021-08-1013:35Karol WójcikOk I found the answer!#2021-12-0907:52amithgeorgeI found this thread via Slack search. Please share the answer!!#2021-12-0914:28Karol WójcikCreate a schema from a -simple-schema
(def proto-schema (-simple-schema {:pred (fn [x] (satisfies? PROTO x)}))
#2021-12-0914:41amithgeorgeThank you! Will try this. :thumbsup:#2021-08-1013:11ikitommicurrently only on start!. I tried to add var-watching to the once-registered vars, but didn't have the skills to do that. start! & stop! should play nice with reloaded repl thou.#2021-08-1013:39ikitommisilly-hacky-var-watching-diy-horror:
(defn plus1
  "adds 1 to number"
  {:malli/schma [:=> [:cat :int] :int]}
  [x] (inc x))

(add-watch #'plus1 ::watch (fn [_ v _ _] (future (println v "=>" (-> v meta :malli/schema)))))

(defn plus1
  "adds 1 to number"
  {:malli/schema [:=> [:cat [:int {:min 0}]] :int]}
  [x] (inc x))
; =prints=> #'user/plus1 => [:=> [:cat [:int {:min 0}]] :int]
#2021-08-1013:39ikitommiif that could be made robust, could just re-define schmatized vars and the clj-kondo + instrumentation would follow instantly.#2021-08-1013:40ikitommiideas welcome (asked on #clojure too)#2021-08-1014:00emccuewell, if it needs to be "scheduled to fetch later" you could have something like#2021-08-1014:03emccue
(defn schedule-to-refresh 
  [var]
  (set! some-flag-another-thread-is-waiting-on true)
  ;; Or
  (.push some-array-blocking-queue-another-thread-is-waiting-on ...))

(add-watch #'plus1 
  (fn [_ v _ _]
    (schedule-to-refresh v)))
#2021-08-1014:04emccueso at least you don't necessarily interact with the future threadpool#2021-08-1014:05emccueand it isn't a race condition#2021-08-1016:33robert-stuttaford@ikitommi thanks for the humanize buffs 🙂#2021-08-1100:08steveb8n@ikitommi I haven’t tested but would the new release be likely to improve this perf issue? https://gist.github.com/stevebuik/e63735d99fca94041120f9b0e25b616d#2021-08-1105:03Ben Slesstried it out of curiosity, looks like it was improved#2021-08-1108:19Karol WójcikWhat is wrong with this schema?
(def logger-pairs-schema
  [:+ [:tuple keyword? string?]])

(defn pairs-conforms!
  {:malli/schema [:=>
                  [:cat [logger-pairs-schema]]
                  nil?
                  ;; [:or [nil? logger-pairs-schema]]
                  ]}
  [pairs]


  )
#2021-08-1108:20Karol Wójcik
:malli.core/invalid-schema {:schema [:+ [:tuple #function[clojure.core/keyword?] #function[clojure.core/string?--5427]]]}
I don't understand why this throws an error
#2021-08-1108:26Karol WójcikThis is thrown from the new reported
(ns dev                                 
  (:require                             
   [malli.dev :as dev]                  
   [malli.dev.pretty :as pretty]))      
                                        
(dev/start! {:report (pretty/reporter)})
#2021-08-1109:11Ben SlessTry not wrapping logger pairs schema in a vector?#2021-08-1110:07Karol WójcikI don't get it.
(m/validate logger-pairs-schema [[:x "something"]]) => true 
@ikitommi isn't it a bug?
#2021-08-1111:26ikitommino, just an extra vector:
(m/schema [:cat [[:+ [:tuple keyword? string?]]]])
; =throws=> :malli.core/invalid-schema 

(m/schema [:cat [:+ [:tuple keyword? string?]]])
; => => [:cat [:+ [:tuple keyword? string?]]]
#2021-08-1111:27ikitommiwe could make the pretty printer capture schema-creation errors too in dev.#2021-08-1111:27ikitommisame result thou, just more fun with colors 🌈 !#2021-08-1112:20Karol WójcikAhh. Thank you @ikitommi :3#2021-08-1117:12emccue
(mg/generate [:cat string? int?])
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:79).
:malli.core/invalid-schema {:schema :cat}
#2021-08-1117:12emccueis there some context i'm missing?#2021-08-1117:12emccuecat doesn't seem to want to be a valid schema#2021-08-1117:20emccuenvm - seem to have had a really old version floating around the classpath#2021-08-1314:04emccueIt also just dawned on me that the metadata schemas mean I can publish a library with malli schemas and no dependency on malli#2021-08-1314:05emccuei have no doubt that was part of the intent and its silly I just realized it, but its still really cool#2021-08-1314:11Noah Bogartthat is cool!#2021-08-1410:02andre.richardsHi, :uuid schema does not produce correct humanized error:
(me/humanize (m/explain :uuid "foo"))
=> ["unknown error"]
uuid? produces this:
(me/humanize (m/explain uuid? "foo"))
=> ["should be a uuid"]
uuid? has an entry in malli.error but :uuid does not, so it looks like an easy fix. Should I log a issue and raise a pull request?
#2021-08-1410:20ikitommiPlease do!#2021-08-1410:40andre.richardsDone: https://github.com/metosin/malli/pull/512 Thanks for Malli!#2021-08-1410:51ikitommimerged, thanks!#2021-08-1411:39VladislavHi! It’s no option to assoc key with parameters (optional etc.) to schema hm for now, i’m right? I’m forced to use merge for that#2021-08-1413:50ikitommi@shishkov61 You can use a vector to assoc full entry, not at computer, but this should work: (mu/assoc :map [:x {:optional true}] :int)#2021-08-1413:56Vladislavi’ll try it. thanks!#2021-08-1420:59Vladislav
(m/validate 
  [:map [[:set string?] number?]] 
  {#{"a"} 1})
=> false
should it be like this?
#2021-08-1421:08Vladislavactually, i don't see no docs about maps with keys which is not static values#2021-08-1421:11Vladislavshould i use :registry specified schemas for keys?:thinking_face:#2021-08-1421:15Vladislavno, guess i'd found it, but it works another way https://github.com/metosin/malli#qualified-keys-in-a-map#2021-08-1421:28Vladislavo! map-of kinda work for that. but it will be complicated if there is no homogeneous keys needed#2021-08-1519:44Asko NōmmHi! I found a interesting issue in CLJS where Malli fails with "Invalid schema" when checking for object? , for example:
(m/validate [:or map? object?] input)
Fails. However whenever I remove the object?, it starts working. How do I validate a JS object with Malli if object? makes Malli fail?
#2021-08-1520:07emccueonly certain predicates are "built in" - i.e. have an implicit mapping to something. map? gets swapped with [:map], effectively#2021-08-1520:08emccueif you want to use a custom predicate you can either use [:fn object?] - which won't have generation semantics if you use that for your project#2021-08-1520:10emccuethere is also a way to make a custom schema key/generator/etc. I just don't know it offhand#2021-08-1520:10emccuehttps://github.com/metosin/malli#custom-schema-types#2021-08-1520:10emccueit is documented here though#2021-08-1520:46Asko NōmmCheers @U3JH98J4R, this solved it perfectly. I didn't know that it swapped the predicates, this makes sense now!#2021-08-1608:19pithylessI'm working with the malli schema functions (the new dev error reporting is nice!) and I seem to be at a crossroads: 1. If I choose to use m/=> at the top-level: a. I need to an explicit dependency on malli (not generally a problem with applications, but this is probably a no-go for libraries) b. I cannot use non-standard registry schemas, since it is evaled at compile time. c. I think even if I could force some preload logic to make sure my global malli registry is updated before other namespaces are loaded, this is both prone to errors and probably would still not work with external tools that work per-file (clj-kondo, etc) 2. If I choose to use :malli/schema metadata: a. I am free to include it optionally without directly requiring malli b. I can use non-standard registry schemas c. The malli.dev/start! does not pick up var changes (need to force a refresh) d. The metadata approach sometimes makes the function feel "verbose". This is a obviously a subjective feeling, but if the specs are verbose and the function body is short, sometimes the separate m/=> looks nicer. Because of the problems with (1), I'm thinking of still focusing on (2). I can workaround (2c) and perhaps (2d) can be mitigated in certain cases with a (def foo-schema [:=> ...]) and then using (defn foo {:malli/schema #'foo-schema} ..) ?#2021-08-1608:19pithylessPosting these observations, in hopes that someone can set me straight on my misunderstanding; or perhaps suggest a better approach? :)#2021-08-1609:12ikitommi• 1b&c -> this should be resolved somehow, please write an issue • 2c -> can be fixed (look up for var changes automatically) • d -> true that what might help if malli could pick up the function schemas from the actual functions behind the vars. it would allow one to say:
(defn kikka [])

;; just clojure
(defn => [v schema]
  (alter-var-root v vary-meta assoc :malli/schema schema))

;; add a function schemas without deps to malli
(=> #'kikka [:=> :cat :any])

;; reading from function
(-> kikka meta :malli/schema) ; => [:=> :cat :any]
#2021-08-1609:40pithylessThat looks promising; I will try it out today and submit an issue later. Thanks!#2021-08-1609:12ikitommi• 1b&c -> this should be resolved somehow, please write an issue • 2c -> can be fixed (look up for var changes automatically) • d -> true that what might help if malli could pick up the function schemas from the actual functions behind the vars. it would allow one to say:
(defn kikka [])

;; just clojure
(defn => [v schema]
  (alter-var-root v vary-meta assoc :malli/schema schema))

;; add a function schemas without deps to malli
(=> #'kikka [:=> :cat :any])

;; reading from function
(-> kikka meta :malli/schema) ; => [:=> :cat :any]
#2021-08-1609:40Tiago Dall'OcaHello there#2021-08-1609:46Tiago Dall'OcaI'm trying to create a parser for converting malli's schemas to DTS files (typescript's type definitions file). One of the situations I'm running into is reading malli's schemas correctly, as they're plain data structures with semantic meaning, so I have to check for some options maps when parsing :map or even cases like https://github.com/metosin/malli#qualified-keys-in-a-map. I was thinking of writing a https://github.com/metosin/malli#parsing-values for helping me out with that, but maybe it already exists? Or is there an easier way?#2021-08-1609:59ikitommiSounds great, looking forward to this! You can as m/children and m/properties from a Schema instance. The first one returns the parsed enty-tuples [key properties value] if that helps.#2021-08-1612:53Tiago Dall'Ocam/children is exactly what I was looking for! Thanks :D#2021-08-1615:03richiardiandreaWill monitor this - we have TypeScript as frontend and indeed sharing the API specs would be super cool 🙂#2021-08-1617:22ikitommiBtw, the JSON Schema converter is a good example of generic transformer of malli->xyz, https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc#2022-08-0422:31richiardiandrea@U4U6BDQTE how far did you get with this project? Do you need/would you accept any help? 😄#2022-08-0422:31Tiago Dall'Ocahey there#2022-08-0422:32Tiago Dall'Ocaat the company I work for we started testing it in production actually#2022-08-0422:32Tiago Dall'Ocathe api is not stable#2022-08-0422:32Tiago Dall'Ocathe code is not pretty#2022-08-0422:32Tiago Dall'Ocabut it works for us haha#2022-08-0422:32Tiago Dall'Ocago check it out at flowyoumoney/malli-ts#2022-08-0422:33Tiago Dall'Ocawe didn't setup a collaboration structure just yet, but I think help is welcome!#2022-08-0422:33Tiago Dall'Oca@U0P7M2VHR#2022-08-0422:34Danny🙌#2022-08-0422:41richiardiandreanice that looks great! I think my company would greatly benefit from it and we'll try to give back 😄#2022-08-0423:08DannyAny idea yet how/where you would use it? 🙃#2021-08-1706:38ikitommiAnalysis about Malli Schema creation performance: https://github.com/metosin/malli/issues/513#2021-08-1708:24Ben Slesswoohoo, more performance#2021-08-1708:34Ben Slesscan you uplooad the raw svgs?#2021-08-1708:36ikitommicleared them up already, sorry.#2021-08-1708:48Ben Slessno biggy, I can repro#2021-08-1708:52Ben SlessBtw, regarding the memoization idea I floated
#?(:clj
   (defn memoize!
     []
     (doseq [v
             [#'-validator
              #'-explainer
              #'-parser
              #'-unparser
              #'-transformer
              #'-walk
              #'-into-schema
              #'-safe-pred]]
       (alter-var-root v memoize))))

#?(:clj (defonce _memoized (memoize!)))
Is this sufficient?
#2021-08-1714:14ikitommiunbounded cache? as Schemas don’t have custom equality defined, all instances are different. That would leak memory, a lot. But, interesting idea. Could use that in malli.dev to swap top-level functions into version that pretty print exceptions.#2021-08-1714:16ikitommiSafe thing might be to put the cache into options, so the user can control it. For schema instances, could be bolted into a registry?#2021-08-1714:18ikitommie.g. a registry that returns cached Schemas instead of IntoSchemas in cases it would benefit from caching, e.g. all immutable schmas like leaves without properties and children: :int could be cached, [:int {:title "wadawoksei, kuvavideo"}] is a bad candidate.#2021-08-1715:30Ben SlessWhat is the largest number of schemas you've seen defined? I wonder if an unbounded cache will be good enough in most cases#2021-08-1717:58ikitommiinfinite. sending [:re #".*"] over the wire will always result in a new schema. Regexs don’t implement equality.
(= #".*" #".*") ; => false
#2021-08-1718:05Ben SlessWould it be worth implementing equality semantics for schemas?#2021-08-1718:06ikitommimaybe, there is malli.util/equals already, but it doesn’t take into account the possible different local registry bindings.#2021-08-1718:06ikitommiI recall there is an issue about pushing the local schema binding into “full”-form…#2021-08-1718:07ikitommiI think any option could effect how the schema works, so the equality might be heavy to calculate, might be wrong.#2021-08-1718:10ikitommibtw, thanks again for you efforts on perf, really appreciate it 🙇#2021-08-1718:10Ben SlessI'm just having lots of fun with it#2021-08-1718:10Ben SlessAnd it's all your fault for giving the perf talk at ClojureTRE 2019#2021-08-1718:11Ben Sless😄#2021-08-1816:03ikitommi#2021-08-1717:17Ben SlessIs it possible to use regex schema to say something like "I don't know what these two consecutive elements are but they should be identical"?, i.e. [1 1 1] would be valid but [1 2 1] would not?#2021-08-1723:59lsenjovSo all elements should be identical?#2021-08-1802:47lsenjovOr just certain positional elements?#2021-08-1804:50Ben SlessI gave a really bad example, sorry Better example, for N=2 and T=int: [1 1 7 7 8 8 2 2 3 3]#2021-08-1804:52ikitommidon't think so, :and would be the way to do this, but it currently pushes you out of the sequence.#2021-08-1804:56ikitommialso noticed that function schemas expect a :cat, which is bad as one can't wrap it like [:and [:catn [:min :int] [:max :int]] [:fn (fn [min max] (< min max))]]#2021-08-1805:43Ben SlessCould it be done with textual regular expressions using capture groups?#2021-08-1806:13Ben SlessSomething like (P)\1#2021-08-1905:51ikitommian inline/hidden :and would be my first guess, something like:
[:* [:and [:repeat {:min 2, :max 2} :int] [:fn (fn [[x y]] (= x y))]]]
#2021-08-1905:52ikitommi:and is already weird, as the first thing is used in utilities. I guess this is the reason why Schema named it constrained. One Schema + constraints. Not “all the schmas”#2021-08-1913:08Ben Slesswas hoping to avoid fn#2021-08-1913:15ikitommiI think it’s the same thing as key-relations for maps, but… sequence relations instead.#2021-08-1913:17ikitommimalli + meander here could look like:
[:* 
 [:and 
  [:repeat {:min 2, :max 2} :int] 
  [:relations 
   '[?min ?max] 
   '[:= ?min ?max]]
#2021-08-1913:19ikitommiand maps:
[:and
 [:map
  [:min :int]
  [:max :int]]
 [:relations 
  '{:min ?min, :max ?max} 
  '[:> ?min ?max]]]
#2021-08-1913:34Ben Slesswith catn it's exactly the same#2021-08-1915:29kennyMalli often prefixes function names with -. Often this implies a private or internal function, yet lots of - functions are used and encouraged. What is the meaning behind the - prefix in the Malli codebase?#2021-08-1915:30emccue"read the docs"/"experts only"#2021-08-1915:34kennyI've read the docs several times and missed that section each time 🙂 Thanks. fwiw, those functions do not seem like "experts only." They have been critical to our use of Malli.#2021-08-2019:43dominicmIs there anything higher level than m/type for determining, e.g. sequences? I'd like to lump :* ,`:+`, :tuple, etc. together for my purposes when doing programmatic work on schemas. Same idea for pos-int? int?, etc.#2021-08-2019:58ikitommi@dominicm currently no, ideas welcome. There is an issue about derived types. int? is actually just a :`int`, silly to have to declare humanized error messages, generators, JSON schema mappings, transformers for both.#2021-08-2019:58ikitommihttps://github.com/metosin/malli/issues/264#2021-08-2020:02dominicm@ikitommi For sure. My use-case is along those lines. I'm trying to determine if a schema is, e.g. valid JSON (without transforms). Pairs nicely with Muuntaja 😉#2021-08-2020:03dominicmI'll have a look at the source of those features for my solution#2021-08-2104:01ikitommiyes, transformers know that, quick hack would be to loop the registered keys of json-transformer or even ask if there is anything to do:
(= identity
   (m/decoder
    schema
    (mt/json-transformer))
#2021-08-2110:50Ben SlessSome perf finding: invoking map on arguments is the shortest path to .valAt without interop. Faster than get#2021-08-2114:24dominicm@ikitommi That doesn't quite work:
(m/decoder
    [:map [{"type" "foo"} string?]]
    (malli.transform/json-transformer))
Which can't be encoded as valid JSON.
#2021-08-2114:26dominicmheh, jsonista converts the key to a edn string in that case: "{\"{\\\"type\\\" \\\"foo\\\"}\":\"bar\"}"#2021-08-2114:46ikitommi@dominicm right, there is no decoders registered for keys in json-transformer, it's expected to come clean from actual JSON decoder. And same for encoding, malli doesn't have JSON encoders for all schemas, as in the actual encoder (jsonista, Cheshire etc) already does that.#2021-08-2116:33Ben SlessOpened a bunch of small PRs relating to https://github.com/metosin/malli/issues/513 . I broke them down as much as possible to find the best performance gains for each change and to cover enough use cases in the benchmarks. Didn't want to end up accidentally slowing down one while speeding up the other (which I did, initially)#2021-08-2211:49anonimitorafHi guys, I'm looking to use malli for the following use cases... Context: • I'm working on a system that takes in data from various 3rd party sources (think scraping). • The "scraped" data get transformed into an internal, universal data format that then get saved to DB Problems to be solved: • Validating that the 3rd party data are of some expected shape Obvious how malli gets utilized here • Mapping the 3rd party to the internal, universal data format Is this possible to be done via custom transformers on each property of the scraped data? I can give examples if needed#2021-08-2211:53Ben SlessDefinitely possible#2021-08-2211:53anonimitorafAwesome! I'll write up some dummy examples#2021-08-2211:53Ben Slessprobably more than one way for how you could do it, too#2021-08-2211:58anonimitorafActually, I'm finding it a bit hard to quickly write up dummy examples. Do you mind linking docs on possible ways to do it @UK0810AQ2?#2021-08-2212:01anonimitorafActually here's an example:
(def source-shape
  [:map
   [:events [:sequential
             [:map
              [:id int?]
              [:desc string?]
              [:details-id int?]]]]
   [:details [:sequential
                [:map
                 [:id int?]
                 [:content string?]]]]])
Sample data:
{:events {:id 1
          :desc "Blah"
          :details-id 11}
 :details [{:id 11
            :content "Blargh"}]}
mapped to
[{:id 1
  :description "Blah"
  :details "Blargh"}]
#2021-08-2212:08Ben SlessShould events be a sequence of maps?#2021-08-2212:15anonimitorafAh whoops yes. I'll correct my example#2021-08-2213:15Ben SlessVery roughly:
(defn index-by
  [f xs]
  (reduce (fn [acc x] (assoc acc (f x) x)) {} xs))

(defn join
  [{:keys [events details]}]
  (let [details (index-by :id details)]
    (reduce
     (fn [acc {:keys [details-id] :as event}]
       (conj acc (-> event
                     (dissoc :details-id)
                     (assoc :details (get (get details details-id) :content)))))
     []
     events)))

(def source-shape
  [:map
   {:decode/fun join}
   [:events [:sequential
             [:map
              [:id int?]
              [:desc string?]
              [:details-id int?]]]]
   [:details [:sequential
              [:map
               [:id int?]
               [:content string?]]]]])

(def data
  {:events [{:id 1
             :desc "Blah"
             :details-id 11}]
   :details [{:id 11
              :content "Blargh"}]})

(m/decode
 source-shape
 data
 (mt/transformer {:name :fun}))
#2021-08-2213:15Ben Slessbut there should be a better day to do it#2021-08-2214:29pithylessIMO, things to be aware of: 1. decode is best-effort and will not throw errors if the validation does not pass 2. the transformation functions are expected to handle bad input without throwing errors 3. I think it's best to split the decode-validate-transform into several steps (each can have it's own validation)#2021-08-2214:30pithylessThis is how I'd approach it:
(def source-shape
  [:map
   [:events [:sequential
             [:map
              [:id int?]
              [:desc string?]
              [:details-id int?]]]]
   [:details [:sequential
              [:map
               [:id int?]
               [:content string?]]]]])


(def destination-shape
  [:sequential
   [:map
    [:id int?]
    [:description string?]
    [:details string?]]])
#2021-08-2214:30pithyless
(def sample-data
  {:events  [{:id         "1"
              :desc       "Blah"
              :details-id "11"}
             {:id         "2"
              :desc       "Blah 2"
              :details-id "12"}]
   :details [{:id      "11"
              :content "Blargh"}
             {:id      "12"
              :content "Blargh 2"}]})

;; Example broken input
(def sample-data2
  {:events [{:id nil
             :desc "Blah"
             :details-id 11}]
   :details [{:id "11"
              :content "Blargh"}]})
#2021-08-2214:30pithyless
(let [raw-data sample-data

        ;; First, normalize input (this is best-effort and does not throw errors).
        ;; This can take care of things like converting strings to integers, because
        ;; the serialization protocol could not represent integers, etc.
        decoded (m/decode source-shape raw-data mt/string-transformer)

        ;; Second, validate that the normalized input matches our expectations.
        ;; NOTE: you can use m/validate instead of m/parse if schema has branching logic,
        ;; but you don't want to take advantage of the extra information at this point.
        ;; NOTE 2: you could use a different, more strict schema at this point.
        parsed (m/parse source-shape decoded)
        _ (when (identical? parsed ::m/invalid)
            (throw (ex-info "Invalid Input" (m/explain source-shape decoded))))

        ;; At this point we've validated input and want to do transformation
        transformed (custom-transform-logic parsed)

        ;; And would be good idea to validate transformation worked...
        _ (when-not (m/validate destination-shape transformed)
            (throw (ex-info "Invalid Transformation" (m/explain destination-shape transformed))))]
    transformed)
#2021-08-2214:32pithylessThe custom-transform-logic could be the approach @UK0810AQ2 mentioned; but if we're doing these kind of map/join transformations, it might be worth it to checkout meander (especially when the cases become more complicated to grok):
(defn custom-transform-logic
  [data]
  (-> data
      (meander/search
        {:events  (meander/scan
                   {:id         ?event-id
                    :desc       ?description
                    :details-id ?details-id})
         :details (meander/scan
                   {:id      ?details-id
                    :content ?details})}
        {:id          ?event-id
         :description ?description
         :details     ?details})
      vec))
#2021-08-2214:32pithyless^ FYI @UR37CBF8D :)#2021-08-2214:33pithylessAlso important to mention I used m/validate m/parse etc., but these should all be replaced with m/validator, m/parser etc. in production code#2021-08-2214:37pithylessI apologize for pasting so much code into slack; here's a gist: https://gist.github.com/pithyless/0be222e1b1b3bca0239a9ca07d1b34c2#2021-08-2217:17schmeeI second the recommendation for Meander, perfect use-case for it: https://github.com/noprompt/meander#2021-08-2223:11anonimitorafThanks for the replies guys! I'll check them out after work 🙂#2021-08-2313:20ikitommiYes, meander is the king in complex transformations and works nicely with malli. Great example @U05476190!#2021-08-2314:39pithylessThanks @U055NJ5CC, I was hoping someone more experienced would confirm if the approach was reasonable.#2021-08-2412:58anonimitorafHey guys, again, thanks for these! I somewhat understand how they work. Follow up question: Is there a way to do it via metadata on the malli's schema fields and I somehow manipulate those metadata? I'm thinking something like
(def source-shape
  [:map
   [:events [:sequential
             [:map
              [:id int?]
              [:desc
               {:custom/map-to :description}  ;; <----- like this
               string?]
              [:details-id int?]]]]
   [:details [:sequential
              [:map
               [:id int?]
               [:content string?]]]]])
?
#2021-08-2218:09Ben SlessWrote a mutable entries parser just to see if it was possible. Eliminated almost all the overhead related to parsing entries. It's a disgustingly ugly direct translation of the code, but it works and cut down about 1us / entry#2021-08-2218:43Ben SlessBehold, my horrible creation
#2021-08-2218:43Ben Slesshttps://gist.github.com/bsless/c21995e5702c2fa6d954312786da197b#2021-08-2219:43ikitommi:smiling_face_with_3_hearts:#2021-08-2219:43ikitommimy current project use case is on cljs-side btw.#2021-08-2219:47Ben Sless😄#2021-08-2219:49Ben SlessI'm just surprised it works For cljs we can do what I started with which was propagating the bindings for every case, it should save some, too#2021-08-2219:17respatializedis there a more idiomatic way of testing two schemas for equality than comparing them with malli.util/to-map-syntax?#2021-08-2219:18respatializedlol, just noticed malli.util/equals, please disregard#2021-08-2219:27Ben SlessCaveat emptor
(mu/equals (m/schema [:re #"hi"]) (m/schema [:re #"hi"]))
false
#2021-08-2219:41ikitommiregexp schemas should be thrown away, could be just a property in :string schema#2021-08-2219:42ikitommi[:string {:pattern "hi"}]#2021-08-2316:59pithylessI'm seeing some weird behavior that may (or may not) be related to the intersection of using defprotocol m/=> and malli.dev/start! where I'm getting a StackOverflow exception when running an instrumented function. If I comment out the m/=> everything works fine. Before I go down a rabbit hole of trying to come up with a minimal repro, does this sound familiar to someone?#2021-08-2321:38winsome@mrdalloca, I see up above that you're exploring using malli to generate typescript definitions! I'm just starting to look into that as well, are you open to sharing what you've learned so far? I'm happy to help#2021-08-2321:42Tiago Dall'Ocaheyy#2021-08-2321:42Tiago Dall'Ocasure! I've progressed quite a bit actually#2021-08-2321:43Tiago Dall'OcaI'm working with a draft of it by now#2021-08-2321:43Tiago Dall'Ocaso I'm not happy with the code organization and naming in general#2021-08-2321:43Tiago Dall'Ocavery early stages#2021-08-2321:43Tiago Dall'OcaBUT#2021-08-2321:43Tiago Dall'OcaI'm already generating type definitions#2021-08-2321:44winsomeFantastic! I've got to hop off for other life things in a minute, but do you mind if I message you later? I'd love to try the basics#2021-08-2321:51Tiago Dall'Oca#2021-08-2321:53Tiago Dall'OcaI don't have time left for today, but we can go into details about this tomorrow :))#2021-08-2321:54Tiago Dall'OcaI wake up at 09:00 UTC#2021-08-2403:31datranI won't be about until 14:00 UTC, but I'll ping you then :thumbsup:#2021-08-2323:00respatializedI'm hitting a wall trying to write a schema walker that recursively transforms :altn to :alt, :catn to :cat, etc. The implementation is straightforward for simple schemas, but I encounter failures when I use :refs and try to transform the registry along with the values - the registry doesn't get passed down when the walker descends into the :ref schemas. I'm not sure about the best way to proceed. Rebinding mr/*registry* with a dynamic registry that contains the top-level schema registry seems expedient, but I don't really understand the registry code that well yet which seems perilous.#2021-08-2323:02respatializedHere's what I've got so far. It fails on the call to mu/to-map-syntax in the inner forms because of an invalid :ref error.#2021-08-2414:03AkizHi, do you know what is happening here? bq2pg.config> (m/validator [:map [:export? boolean?] [:import? boolean?] [:gcs-name string?]])
#function[malli.core/-map-schema/reify/reify--9403/fn--9419]
But using > 3 elements fails… bq2pg.config> (m/validator [:map [:export? boolean?] [:import? boolean?] [:gcs-name string?] [:extra string?]])
Execution error (NoClassDefFoundError) at malli.impl.util/f (util.cljc:54).
malli/impl/util$f__7020__7021$fn__7045
#2021-08-2414:45Akiz@zikajk This happens only when I use malli.clj-kondo +
(-> (mc/collect *ns*) (mc/linter-config))
(mc/emit!)
(md/start!)
after some time… I can not reproduce it well enough 😞
#2021-08-2421:07niclasnilssonI’m trying out the instrumentation, and I’m probably doing something wrong, but I just can’t see it right now. The simple cases with primitives from the docs works, but when I try to validate a more “normal” schema, I can’t get it to work. This is an example:
(def coord-schema
  [:map
   [:x :int]
   [:y :int]])


(defn down
  {:malli/schema [:=> [:cat coord-schema] [coord-schema]]}
  [coord]
  (update coord :y dec))
When I then do (dev/start!) I get the error
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:119).
:malli.core/invalid-schema {:schema [:map [:x :int] [:y :int]]}
What am I missing?
#2021-08-2421:55niclasnilssonLooks like I was missing [:schema …]. This works.
(defn down
  "Some docstring"
  {:malli/schema [:=> [:cat [:schema coord-schema]] [:schema coord-schema]]}
  [coord]
  (update coord :y dec))
#2021-08-2506:19AkizDo you have to use schema when you do not use instrumentation?#2021-08-2506:38ikitommi@U48DE3SHM it should be:
(defn down
  {:malli/schema [:=> [:cat coord-schema] coord-schema]}
  [coord]
  (update coord :y dec))
e.g. don’t wrap schema in empty vector, invalid malli syntax.
#2021-08-2510:41niclasnilssonAh, so my initial problem was the vector on the output all along. I see. Thanks, @U055NJ5CC.#2021-08-2510:43niclasnilsson@zikajk, I don’t know, but my goal was to use the devevelopment instrumentation, using (dev/start!)#2021-08-2510:59ikitommithe error message was really bad btw: `
malli.core/invalid-schema {:schema [:map [:x :int] [:y :int]]}
could have been something like:
malli.core/invalid-schema {:type [:map [:x :int] [:y :int]]
                           :schema [[:map [:x :int] [:y :int]]]}
#2021-08-2511:51Akiz@U48DE3SHM I experienced two different errors during schema validation when using instrumentation. Restarting helped though…#2021-08-2605:49niclasnilsson@zikajk, ah, thanks, I’ll keep that in mind going forward.#2021-08-2506:40ikitommi@zikajk looks weird. md/start! btw calls the clj-kondo too, so just md/start! and you are done#2021-08-2506:40ikitommiif you can repro, I can take a look#2021-08-2507:36Akiz@ikitommi I will try, I experienced this after long running repl session so there could be a lot of reasons. After restarting - everything is working great.#2021-08-2506:42ikitommiworks here:
(def valid?
  (m/validator
    [:map
     [:export? boolean?]
     [:import? boolean?]
     [:gcs-name string?]
     [:extra string?]]))

(valid? 
  {:export? true
   :import? true
   :gcs-name "kikka"
   :extra "foo"})
; => true
#2021-08-2506:55Karol WójcikI'm trying to use reitit-malli coercion with malli 0.6.1. When evaling [reitit.coercion.malli :as rss] I'm getting
------ WARNING - :fn-arity -----------------------------------------------------
 Resource: <eval>:2007:43
 Wrong number of args (3) passed to malli.core/-fail!
--------------------------------------------------------------------------------
Can someone reproduce?
#2021-08-2507:15ikitommi@karol.wojcik that’s a bug, the 3-arity was removed. please write an issue, need to fix reitit (or revert the change if it’s really needed)#2021-08-2507:30Karol WójcikDone: https://github.com/metosin/reitit/issues/504#2021-08-2507:27Karol WójcikThank you @ikitommi for blazing fast response. I will write an issue! No worries I can wait 🙂#2021-08-2508:16ikitommiEveryone: malli master has massive performance improvements related to Schema creation and transformation performance. If you are using malli especially with cljs, you might see order of magnitude improvements to startup-times. Please try and report both gains and bugs. Big thanks for @ben.sless for much of the hard work.
(def ?schema
  [:map
   [:x boolean?]
   [:y {:optional true} int?]
   [:z [:map
        [:x boolean?]
        [:y {:optional true} int?]]]])
        
;; 44µs => 8.5µs (5x)
(def schema (m/schema ?schema))

;; 26µs => 1.3µs (20x)
(m/walk schema (m/schema-walker identity))

;; 51µs => 6.5µs (8x)
(mu/closed-schema schema)
… there is still a lot of room for improvements, but would like to release these soon. Issue tracking, alternative ways to build large schema systems described and comment welcome on https://github.com/metosin/malli/issues/513.
#2021-08-2510:20Ben Sless@ikitommi Here's a riddle I can't find a good answer to, how come this is significantly faster than a PersistentHashMap backed registry
(defn fast-registry
  [m]
  (let [pred (comp keyword? key)
        im (doto (new IdentityHashMap 1000) (.putAll (into {} (filter pred) m)))
        -m (doto (new HashMap 1000 0.25) (.putAll (into {} (remove pred) m)))]
    (reify mr/Registry
      (-schema [_ k]
        (let [^Map m (if (instance? Keyword k) im (if (instance? Var k) im -m))]
          (.get ^Map m k)))
      (-schemas [_] m))))
By about 35 ns for keyword lookup compare
(cc/quick-bench (m/schema :int {:registry r}))
  (cc/quick-bench (m/schema :int))
But the registry lookup isn't that much faster, only ~7 ns
(cc/quick-bench (mr/-schema r :int))
  (cc/quick-bench (mr/-schema m/default-registry :int))
#2021-08-2510:50ikitommi🤷 micro benchmarks are hard. I would ask JMH how they really differ. Just swapping order of the tests, you might get different results. Also, more code, JIT behaves differently.#2021-08-2510:51ikitommiSuper quick googling: https://www.oracle.com/technical-resources/articles/java/architect-benchmarking.html#2021-08-2512:12Ben SlessInteresting, JMH gives opposite results#2021-08-2511:03lreadThe malli readme might be a good candidate for https://github.com/lread/test-doc-blocks. I could take a whirl at it if there is any interest.#2021-08-2520:43lread@ikitommi I won’t proceed unless you indicate an interest. Might be your cup of tea and might not! Happy to do the PR for you to evaluate then decide if you like - or not.#2021-08-2511:55martinklepschI’m getting an fn-arity warning when requiring malli.core in a cljs repl:
; eval (current-form): (require 'malli.core)
; (err) ------ WARNING - :fn-arity -----------------------------------------------------
; (err)  Resource: <eval>:2007:43
; (err)  Wrong number of args (3) passed to malli.core/-fail!
; (err) --------------------------------------------------------------------------------
#2021-08-2516:42Karol WójcikHad the same issue. It's fixed in master#2021-08-2517:56martinklepschthanks Karol!#2021-08-2520:03Karol WójcikYou welcome Martin! 🙂#2021-08-2613:34martinklepsch@ikitommi do you think it’s worth documenting this in the changelog for 0.7?#2021-08-2614:13ikitommiPR welcome @U050TNB9F, but there is a a already this line: > * fixed arity error in `m/function-schema`#2021-08-2614:13ikitommidoes not pinpoint the problem very well :thinking_face:#2021-08-2512:37AkizHi, I am getting Expected: regular expression, received: string. in clj-kondo lsp / Emacs combo with instrumentation for : (def gcs-uri "") (gcs/get-blob-moddate gcs-uri) and these are my schemas:
(m/=> get-blob-moddate [:=>
                         [:cat Gcs-uri]
                         [:or
                          [:fn #(= (type %) org.joda.time.DateTime)]
                          nil?]])
(def Gcs-uri
  [:re #"gs:\/\/.*"])
#2021-08-2516:42Karol WójcikIs it possible to instrument function via metadata in Clojurescript?#2021-08-2517:09ikitommisomeone needs to port the tooling to support cljs. I suck at macro+interop, so not going to do that any time soon. help most welcome.#2021-08-2517:10Karol WójcikI will take a look into it when I have some time 🙂#2021-08-2517:22lreadAm having a great time experimenting with malli to validate cljdoc’s cljdoc.edn . The docs and demos are great in helping me to understand how things work. Thank you for all that effort. As a newb, I was wondering about the following: 1. The https://malli.io/?value=%5B1%20%5B2%20%5B3%20%5B4%20nil%5D%5D%5D%5D&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22ConsCell%22%20%5B%3Amaybe%20%5B%3Atuple%20%3Aint%20%5B%3Aref%20%22ConsCell%22%5D%5D%5D%7D%7D%0A%20%22ConsCell%22%5D uses [:ref "ConsCell"] but the https://malli.io/?value=%5B%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D%5D&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D uses [:schema [:ref "hiccup"]]. Are [:ref x] and [:schema [:ref x]] equivalent? 2. I was unfamiliar with :maybe and was not confident what it did. From the examples it seems that [:maybe x] validates for x or nil? 3. I was unfamiliar with :tuple and now think I understand it just fine. But maybe a sentence contrasting it with :vector (if that makes sense) would be helpful to newcomers. 4. Very subjective, but I would put the motivation section before examples. As a newbie, this kind of info would be helpful to me up front. If updates to docs and demos for any the above would be helpful I am happy to make those wee changes. Also once I’ve finalized my cljdoc.edn validation I would be happy to add that as a, perhaps familiar, example to docs, demos or both.#2021-08-2517:46Ben SlessI think schema of ref cannot be derefed. This lets you write things like recursive sequence schemas#2021-08-2518:08lreadThanks @ben.sless! - Are you suggesting maybe the hiccup demo might be a bit off?#2021-08-2518:09Ben SlessCan't say without giving it a deeper look. I bumped by head against it separately trying to port the datalog BNF to malli#2021-08-2518:10Ben SlessSchema schemas create a "hard" boundary when schemas are compiled and you don't get deref-ed recursively ad inifinitum#2021-08-2518:10Ben Slessyour understanding of :maybe is correct#2021-08-2518:11Ben Slesswith touple, just making sure, it is a heterogeneous fixed length product type#2021-08-2518:12lreadis a tuple necessarily a vector?#2021-08-2518:12Ben SlessIt seems so#2021-08-2518:13lreadCoolio, so I think I get it and could maybe add a sentence in the README.#2021-08-2518:13Ben SlessI'm working on a getting started guide 🙃#2021-08-2518:14lreadOh cool! Maybe I don’t need to update README then.#2021-08-2518:14Ben SlessIt's a good idea to do so anyway, it's a small change you can do now, the guide will take some time#2021-08-2518:14lreadPing me if you want a reviewer. I’m a newb!#2021-08-2518:15Ben SlessI have a bunch of guinea pigs at work 😛#2021-08-2518:16Ben SlessBut once it's sorta baked I'll go hunting for external perspectives#2021-08-2518:16Ben Slessso expect a ping in your future!#2021-08-2518:16Ben SlessJust, you know, don't deref it or anything#2021-08-2518:19lreadWhen working on your datalog BNF were you able to generate and sample without sporadic StackOverflowError exceptions?#2021-08-2518:20Ben SlessYes, let me check if it's public I'll send it#2021-08-2518:22lreadtx#2021-08-2518:23Ben Slesshere you go https://github.com/bsless/mallilog/blob/master/src/mallilog/impl/schema.clj#2021-08-2518:24Ben SlessNote the abundance of [:schema [:ref ,,,]]#2021-08-2518:26lreadYeah! So maybe that tells us something about my first question(?). You don’t seem to use :ref without it wrapped in :schema.#2021-08-2518:28Ben SlessThat's because my schemas are mostly recursive but I probably don't need to do it everywhere#2021-08-2518:29Ben SlessI can try by process of elimination to knock it out and see what's up but it's pedantic and didn't really feel like fighting with it#2021-08-2518:29Ben Slessbesides, I consider it a bit of an anti-feature#2021-08-2518:43lreadthe having to do something special for the recursive case?#2021-08-2518:43Ben SlessYeah, I'd expect a ref to be enough#2021-08-3119:47lreadTook a stab at https://github.com/metosin/malli/pull/537.#2021-08-2521:22lreadHiya mallians! I’m making some good progress with my cljdoc.edn malli schema. I am, though, getting some StackOverflowError exceptions when generating samples. This might be normal, I’m new, what do I know? simple_smile Rather than share and diagnose my cljdoc.edn schema, it might be more fruitful to work from the familiar. The Hiccup schema from the malli README seems to have-ish the pattern of recursion I want for cljdoc.edn. If I take the Hiccup schema unchanged from the README I do not seem to suffer StackOverflowError exceptions.
(def Hiccup
  [:schema {:registry {"hiccup" [:orn
                                 [:node [:catn
                                         [:name keyword?]
                                         [:props [:? [:map-of keyword? any?]]]
                                         [:children [:* [:schema [:ref "hiccup"]]]]]]
                                 [:primitive [:orn
                                              [:nil nil?]
                                              [:boolean boolean?]
                                              [:number number?]
                                              [:text string?]]]]}}
   "hiccup"])

(mg/sample Hiccup)
But if I yank out some stuff and make the schema look a little more like my cljdoc.edn schema usage, I can easily trigger StackOverflowError exceptions by re-evaluating (mg/sample Hiccup2) a handful of times:
(def Hiccup2
  [:schema {:registry {"hiccup" [:catn
                                 [:name keyword?]
                                 [:props [:? [:map-of keyword? any?]]]
                                 [:children [:* [:schema [:ref "hiccup"]]]]]}}
   "hiccup"])

(mg/sample Hiccup2)
I am likely quite naive in my understanding here. I’m guessing if I somehow limited the maximum tree depth (which would be reasonable thing to do anyway for my use case) that would help but I am not sure how to express that. I am using malli 0.6.1.
#2021-08-2611:15jeroenvandijk@UE21H2HHD Without saying i’m an authority here, it seems that your schema doesn’t have a stop condition? So endless recursion? Whereas the hiccup one has an alternative which stops the recursion. Makes sense?#2021-08-2611:15jeroenvandijkThe only stop condition you have is when there are 0 children for one node#2021-08-2614:42lreadAh right… thanks @U0FT7SRLP! That makes perfect sense. For the generator, the :orn is probably a 50/50 coin toss for termination whereas hitting 0 children for termination is much less likely.#2021-08-2614:46jeroenvandijkYeah and then there is the branching of the children 😅 1 child still ok, but 2 or more and there you go#2021-08-2614:48lreadSo constructing a schema for validation is one thing and then adapting it to to support sensible generation is another consideration.#2021-08-2614:49lreadOr maybe folks write custom generators?#2021-08-2615:00lreadOh I see :gen/* stuff can influence generation, I’ll poke around more.#2021-08-2616:02jeroenvandijkWith clojure.spec https://clojure.org/guides/spec#_custom_generators, so I’m guessing this would make sense for Malli too. (I haven’t done it yet with Malli)#2021-08-2712:03martinklepschHow would I express that a schema can only have key :a OR key :b but not both? #2021-08-2713:09lreadWould something like this work?:
[:and
   [:map
    [:x {:optional true} int?]
    [:y {:optional true} int?]]
   [:fn {:error/message "Only one of :x or :y allowed"} (fn [{:keys [x y]}] (not (and x y)))]]
#2021-08-2713:14lreadIn https://malli.io/?value=%7B%3Ax%201%20%3Ay%202%7D&amp;schema=%20%5B%3Aand%0A%20%20%20%5B%3Amap%0A%20%20%20%20%5B%3Ax%20%7B%3Aoptional%20true%7D%20int%3F%5D%0A%20%20%20%20%5B%3Ay%20%7B%3Aoptional%20true%7D%20int%3F%5D%5D%0A%20%20%20%5B%3Afn%20%7B%3Aerror%2Fmessage%20%22Only%20one%20of%20%3Ax%20or%20%3Ay%20allowed%22%7D%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(not%20(and%20x%20y)))%5D%5D#2021-08-2713:59ikitommithere are ideas around declarative way of doing that in the future, either using datascript or meander syntax.#2021-08-2713:31mike_ananevHello! I'm trying to use an example from malli docs. Why I got an error?
(defn plus1 [x] (inc x))
(m/=> plus1 [:=> [:cat [:int {:max 5}]] [:int {:max 6}]])
;; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:119). :malli.core/invalid-schema {:schema :=>}
#2021-08-2913:48mike_ananevhmm, I see a strange behaviour. If I run in terminal repl, everything woks fine. If I run via Idea+Cursive then I see this error. Btw, I deleted .cpcache completely.#2021-08-2913:50mike_ananevIn Emacs I see the same error#2021-08-2914:20mike_ananev@U055NJ5CC I've found the reason of that error. If I add `
:jvm-opts    ["-Dmalli.registry/type=custom"]
to deps.end alias, then I see this behaviour. Is it normal?
#2021-08-2916:17mike_ananevfound solution for this: `
(mr/set-default-registry! (m/default-schemas))
#2021-08-2916:58ikitommigood to hear you got it reaolved#2021-08-2713:31mike_ananevmalli version 0.6.1#2021-08-2714:16Brett RowberryI think the answer to this question is probably going to be "just use reitit". Anyway, compojure.api has support for coercion and validation via Schema and spec. Are there examples on how to use malli instead?#2021-08-2714:22Tiago Dall'OcaMALLI-&gt;TS - a sneak peek#2021-08-2806:38anonimitorafHi guys (kinda new to malli), when walking the schema, how would I walk the schema and transform the value of each node with custom logic? I've looked at the src of the json-transformer, etc but they go over my head at the moment 😞 e.g.
(def my-schema [:map 
                 [:a [string? {:custom/upper}]]
                 [:b [int? {:custom/square}]]])

(def my-data {:a "a", :b 2})

;; Transform, key :a to be upper-cased and :b squared
{:a "a", :b 5} => {:a "A", :b 25} 
#2021-08-2808:59pithylessBut that's just my opinion; perhaps other #malli users will weigh-in with their own perspectives :)#2021-08-3115:13olynice I came to ask pretty much this, I have a malli spec which takes keywords and goes through the default json transformer and failes because its a string, looks like I can just add :decode/json {:leave keyword} to the spec and it should transform to the correct format#2021-08-2808:16pithylessHmmm, the default interceptor transformers (which I admit I have not tried to customize) do not behave as I would have expected:
(m/decode [int? {:decode/json {:leave 'inc}}] 5 (mt/json-transformer))
  ;; => 6

  (m/decode [int? {:decode/json {:leave 'inc}}] "5" (mt/json-transformer))
  ;; (expected) error! int is a valid json value, so should not have been represented as string

  (m/decode [int? {:decode/string {:leave 'inc}}] 5 (mt/string-transformer))
  ;; => 6
  
  (m/decode [int? {:decode/string {:leave 'inc}}] "5" (mt/string-transformer))
  ;; (unexpected) error! I would have expected the default string enter/leave interceptors to run by now, so we can `(inc 5)
`
#2021-08-2808:59pithylessBut that's just my opinion; perhaps other #malli users will weigh-in with their own perspectives :)#2021-08-3008:36andre.richardsIs there a way to pass custom registry to function schemas (metadata function schemas specifically)? I.e. when calling m/validate you can specify :registry key in options map, but not sure how to do this for functions. I tried to add :registry to options map passed to m.dev/start! but that seems to be ignored.#2021-09-0114:53CélioIs there a way to customize the “missing required key” error message? For example, the code below returns the default missing key error message instead of something's wrong which makes sense because :error/message relates to the contents of :something, not to whether the key is present. It would be great if we could customize missing key errors.
(me/humanize (m/explain
              [:map
               [:something
                [:string {:error/message "something's wrong"}]]]
              {}))
#2021-09-0115:16dergutemoritzHave you tried putting it on the entry's props?#2021-09-0115:16dergutemoritzas in:
[:something {:error/message "something's wrong"} :string]
#2021-09-0217:01Célio@U06GVE6NR Yes, the result is the same.#2021-09-0118:06kirkedWhat's the best way to specify a schema for a map with both fixed keys and dynamic keys? Conceptually what I'd like is something like this schema to specify that a step has a :run keyword and some dynamic keys, and a workflow has :name and :start keywords and some dynamic keys:
(def step
  [:and [:map [:run :qualified-keyword]]
        [:map-of :keyword [:map [:next :keyword]]]])

(def workflow
  [:and [:map [:name :string]
              [:start :keyword]]
        [:map-of :keyword step]])
The data has this shape:
{:name "my-workflow"
 :start :begin

 :begin
 {:run :app/init
  :ok {:next :do-work}
  :fail {:next :done}}

 :do-work
 {:run :app/process-queue
  :ok {:next :done}}

 :done
 {:run :app/cleanup}}}
The problem I've run into is that I need to #(apply dissoc % fixed-keys) before the :map-of schema is evaluated for any map matching this pattern. Maybe there's a better way?
#2021-09-0817:20ikitommisorry, no, but there is an issue for that: https://github.com/metosin/malli/issues/43 @U09EGLJLB#2021-09-0211:21borkdude@ikitommi Congrats on the CT funding!#2021-09-0213:11ikitommithanks! and congrats on you @borkdude too on the full year! (the application on the page is the old one, those are all done, send email to update that)#2021-09-0604:59emccueremembering typescript has https://github.com/DefinitelyTyped/DefinitelyTyped and wondering if it would be sensible to do something similar for clj* libraries and malli#2021-09-0605:00emccueMotivating example for me most recently is datomic's api - it gives annoying and obtuse errors if you pass bad data and spec'ing inputs at dev time would be helpful#2021-09-0605:01emccue
(alter-meta!
  #'d/transact
  assoc
  :malli/schema
  [:=>
   [:cat some? [:map [:tx-data some?]]]
   [:map
    [:db-before any?]
    [:db-after any?]
    [:tx-data  any?]
    [:tempids any?]]])
#2021-09-0605:03emccueidk what the best approach is - maybe artificially associate with a namespace at first instead of the metadata?#2021-09-0605:05emccue
(defn register-possibly-nonexistant-schema
  [var-path schema]
  (m/-register-function-schema! (symbol (namespace var-path))
                                (symbol (name var-path))
                                schema
                                nil))

(register-possibly-nonexistant-schema
  'datomic.client.api/transact
  [:=>
   [:cat some? [:map [:tx-data some?]]]
   [:map
    [:db-before any?]
    [:db-after any?]
    [:tx-data  any?]
    [:tempids any?]]])
#2021-09-0605:05emccueor maybe just ignore it and let library authors do it themselves#2021-09-0605:05emccueand fight the inevitable war with whatever is finally christened spec 2#2021-09-0605:13emccueor maybe the library has its own registry you can select from and merge into your own / the global registry#2021-09-0605:13emccueidk#2021-09-0605:17lsenjovPick a top level namespaces that does all the (m/=> ...) declarations, and they can include it themselves if they feel so inclined?#2021-09-0617:17Noah BogartClj-kondo has this for linting. Maybe malli could adopt a similar pattern? #2021-09-0617:31ikitommithat would be interesting. One of my goals with clojurists-together is to get a robust schema inferrer: for runtime for inferring from schemas from samples, but could be used to pull out schemas from tests (annotate functions in dev-mode to infer input & output) schemas. Not a silver bullet, but would be “for free”. Manually writing functios schmas is always better, but also, a lot of work.#2021-09-0812:02aaron51Is there a malli helper that converts a map to a := schema? Input is {:a 1 :b 2} Output is [:map [:a [:= 1]] [:b [:= 2]]] I’m trying to validate a subset of a map (and the input is the sub map)#2021-09-0812:06ikitommi@aaron51 not yet. but plan is to build both a first-class inferrer and re-visit the type hierarchies so we could narrow down things. before that, you could hack the malli.provider, it’s quite small and relatively simple.#2021-09-0812:23aaron51Understood, thank you 👍#2021-09-1211:07Ben SlessThis is somewhere between malli and #reitit, is there a way to know the schema with which validation passed in cases of or and multi?#2021-09-1214:53respatializedI asked this question a while back; there's not a good way to do this without orn. If you know all the validation paths at compile time they can be enumerated. But the use case I was thinking about, of sum types that are created at runtime, was still tricky to match on. @U055NJ5CC suggested a simple function for converting the or to orn with numbers as the keys so that I could know which was matched. Not sure if anything has changed in the library since I asked about it, so this solution may be out of date.#2021-09-1214:57Ben SlessEnded up going with orn and parse in the end I wonder how it interacts with decoding. Are conflicts possible?#2021-09-1218:11rutledgepaulvI've used a custom decoder for this that attaches the schema to the value as metadata#2021-09-1620:13Yehonathan SharvitAre there some tools out there that generate a data model diagram (in a format like DOT or PlantUML) from a JSON Schema?#2021-09-1712:10emccuemalli can make uml from its own schema#2021-09-1820:32Yehonathan SharvitI know. But my question is about JSON schema#2021-09-1820:32Yehonathan SharvitIs there a way to transform a JSON schema into a Malli schema?#2021-09-1820:58emccueMaybe an equivalent question is "are all things representable in JSON schema representable in malli"?#2021-09-1820:59emccueBecause i don't think anyone has done that transformation yet#2021-09-1821:10nivekuilcould you infer a malli schema from the corresponding json object?#2021-09-1916:24ikitommiI noticed you got already some answer on Twitter @U0L91U7A8, could you share those here too?#2021-09-1917:29Yehonathan SharvitUnfortunately, none of the answers were satisfactory.#2021-09-1917:30Yehonathan SharvitThe best I could do is to manually convert the JSON schema into Malli and use Malli to generate a plantuml diagram#2021-09-1917:32ikitommiOk. The new map-syntax is quite close to JSON Schema, which should make the JSON Schema -> Malli streightworward.#2021-09-2208:29Yehonathan SharvitIndeed. @U055NJ5CC What was the motivation for adding the new map-syntax?#2021-09-2210:24ikitommi1. order of magnitude faster way to instantiate schemas (e.g. ast IS the parsed results, no need to parse hiccup) -> matters on js & mobile 2. the old map-syntax was too generic to be useful 3. creating schemas programmatically behind the scenes is easier with maps (e.g. when inferring schemas from data) 4. it’s good to have options 😎#2021-09-2215:01Yehonathan SharvitMakes sense. Thanks#2021-09-1820:32Yehonathan SharvitIs there a way to transform a JSON schema into a Malli schema?#2021-09-2210:24ikitommi1. order of magnitude faster way to instantiate schemas (e.g. ast IS the parsed results, no need to parse hiccup) -> matters on js & mobile 2. the old map-syntax was too generic to be useful 3. creating schemas programmatically behind the scenes is easier with maps (e.g. when inferring schemas from data) 4. it’s good to have options 😎#2021-09-1914:30WonderLastkingHi there, i'm using malli 0.6.1 to do some validation in a front-end (cljs)/back-end(clj) application. So I have a .cljc namespace where I do the validations. While this works right with clj, when I use it in cljs I see a warning message
Wrong number of args (3) passed to malli.core/-fail!
#2021-09-1916:34ikitommifixed in master#2021-09-1916:28ikitommiclosed the performance Issue (https://github.com/metosin/malli/issues/513) - schema creation & transformation is now mostly an order of magnitude faster:
(def schema
  [:map
   [:x boolean?]
   [:y {:optional true} int?]
   [:z [:map
        [:x boolean?]
        [:y {:optional true} int?]]]])

(def schema (m/schema ?schema))

;; 44µs -> 3.4µs (13x)
(bench (m/schema ?schema))

;; 4.2µs -> 830ns (4.5x)
(bench (mu/assoc schema :w :string))

;; 134µs -> 15µs (9x)
(bench (mu/merge schema schema))

;; 51µs -> 3.9µs (13x)
(bench (mu/closed-schema schema))
#2021-09-1916:31ikitommi… and opened a new one, to add first-class support for the (compact) map-syntax: https://github.com/metosin/malli/issues/543. This is really important in for building large schema systems a) via inferring or b) to be run on slow js-runtimes. Initial design (which is already 15x faster 🚀)
(def ?schema
  [:map
   [:x boolean?]
   [:y {:optional true} int?]
   [:z [:map
        [:x boolean?]
        [:y {:optional true} int?]]]])

(m/form ?schema)
;[:map
; [:x boolean?]
; [:y {:optional true} int?]
; [:z [:map
;      [:x boolean?]
;      [:y {:optional true} int?]]]]

(m/ast ?schema)
;{:type :map,
; :keys {:x {:order 0
;            :value {:type boolean?}},
;        :y {:order 1, :value {:type int?}
;            :properties {:optional true}},
;        :z {:order 2,
;            :value {:type :map,
;                    :keys {:x {:order 0
;                               :value {:type boolean?}},
;                           :y {:order 1
;                               :value {:type int?}
;                               :properties {:optional true}}}}}}}

(-> ?schema
    (m/schema) ;; 3.7µs
    (m/ast) ;; 1.1µs
    (m/schema) ;; 250ns (15x)
    (m/form)
    (= (m/form ?schema)))
; => true
#2021-09-1916:34ikitommiit’s still fully generic, like the mu/to-map-syntax & mu/from-map-syntax , but using a new support protocol, schemas can do whatever they want. The new ast can be used as “I know what I’m doing” kinda way => no parsing & checks needed.#2021-09-2016:04ikitommi
(assert
  (= (m/form [:=> [:cat :int] :int])
     (m/form {:type :=>
              :input {:type :cat
                      :children [{:type :int}]}
              :output {:type :int}}))
  "malli supports both hiccup & map syntax")
#2021-09-2016:06ikitommihttps://github.com/metosin/malli/pull/544/commits/e44f8c027699a63f73b941137151e5a489501587#2021-09-2107:35eskosIs this unified in the sense that all hiccup gets internally turned into maps and then whatnot or two separate paths? Just trying to think onwards whether this could actually be a cause of sneaky conversion bugs in the long run…#2021-09-2107:36eskos(I haven’t looked into malli internals in ages so judging a single commit diff doesn’t answer that to me)#2021-09-2205:30ikitommithere is just one path, currently the map-syntax (the new ast) is transformed into properties children and thus, the old code path is used.#2021-09-2205:32ikitommiI’ll most likely reverse that, so the default path is the AST, just a hiccup->ast converter is just mechanical.#2021-09-2205:34ikitommiwhy? the ast-path is much faster (with maps), as there is no parsing needed. if real life projects, seen 10x diffrence, 100ms vs 1sec to load all the hundreds of schemas in a slow device.#2021-09-2205:35ikitommialso, the example was broken, the conversion was nor recursive, fixed inline.#2021-09-2306:12Ben SlessI know there's a lot of focus on performance at the moment but what's the status of json-schema and the missing bits, especially date/time?#2021-09-2308:36ikitommigood question. • JSON Schema -> malli, have been waiting for the PR to progress, might be stalled • Date/time, @henryw374 might have an insight on the new js/temporal lib status. If there would be a good set of abstractions for clj/s, malli should use them. Or, just a new alpha ns from the existing clj-time things • others: recursive generation, proper inferring, schematized defn, typescript-compat, map+map-of, local registries refined, ... lot of things to do, some things are started by various contributors, many just ideas. After the ast-cleanup, have clojurists-together -time to work on many things, but we (all) could do design for the missing features and mark them as "PR welcome".#2021-09-2311:37Ben SlessI hope you don't mind, I went ahead and opened a draft PR to add time schemas in clj. I think it's 90% there and I'd rather have 90% to work with vs. 100% in time t > T#2021-09-2311:38Ben Slesslooking at json-schema -> malli gave me a headache, because the schema is versioned! do you want to support "everything" or just latest?#2021-09-2311:39Ben SlessThis has real implications, for example, when I was wrangling vega in #datavis malli could have been a great help but vega uses an old version of json schema#2021-09-2308:39ikitommiI tried to categorize the issues, but could add more explicit tags on those that are discussed already and would be ready to be picked up for anyone wanting to contribute.#2021-09-2308:40ikitommithe ast-change is fundamental, planning to finish a first version of that now. Few days of work I would guess, mainly tests.#2021-09-2311:41CarloI know of aave and snoop for instrumentation, is there a library in the malli ecosystem that does automatic generative testing of functions?#2021-09-2418:49Tiago Dall'OcaHow to get the original form of a schema-id? Details in the thread#2021-09-2418:49Tiago Dall'Oca#2021-09-2418:50Tiago Dall'Oca#2021-09-2418:50Tiago Dall'OcaIt's strange to me that m/form works for unqualified keys#2021-09-2418:50Tiago Dall'OcaHow do I get this to work for qualified keys too?#2021-09-2419:06Tiago Dall'Oca#2021-09-2419:10Tiago Dall'OcaOk, some more source code reading did it haha#2021-09-2419:11Tiago Dall'Ocaoptions for adding malli schemas to jsdoc#2021-09-2512:17ikitommilooks good imo. what is the status of the ts-mappings @mrdalloca? Is there something you need help with?#2021-09-2513:03Tiago Dall'OcaI'm currently focused on function typing and figuring how to make malli's functions (explain, validate, etc) available to js consumers#2021-09-2513:04Tiago Dall'OcaThen there's some refactoring to do, to make the source code more presentable#2021-09-2513:05Tiago Dall'OcaExternal types (from other libs) will soon become a priority#2021-09-2602:11Carlousing malli/core, is there a way to spec a plus function over positive numbers, asserting that that the result is >= of any of the arguments?#2021-09-2610:14Carlothe closest thing I saw is using the :fn spec to relate the arguments, but I'm not sure how to involve the result of the function#2021-09-2611:00lsenjovI’m not sure you actually can, off the top of my head. You could use pre-post maps in core clojure to test these contracts#2021-09-2611:02lsenjov(If there’s a way to do it I’m sure someone will have a peek when the working week starts)#2021-09-2611:23ikitommi@UA7E6DU04 no, there is no :fn, like there is for spec:
(s/fdef ranged-rand
  :args (s/and (s/cat :start int? :end int?)
               #(< (:start %) (:end %)))
  :ret int?
  :fn (s/and #(>= (:ret %) (-> % :args :start))
             #(< (:ret %) (-> % :args :end))))
, but I don’t see any reason why it would not be supported.
#2021-09-2611:24ikitommiplease write an Issue, so it’s on the backlog.#2021-09-2611:25ikitommialso. there is silly assert inside malli.core that the :input (`:args` in spec) need to be a :cat, so :and doesn’t work. Easy to fix.#2021-09-2611:32ikitommiast-options for eager references, this:
[:and
 {:registry {::a ::b
             ::b ::c
             ::c [:schema pos-int?]}}
 ::a ::b ::c]
ast options: 1️⃣ current:
{:type :and,
 :children [{:type ::m/schema, :value ::a}
            {:type ::m/schema, :value ::b}
            {:type ::m/schema, :value ::c}]
 :registry {::a {:type ::m/schema, :value ::b}
            ::b {:type ::m/schema, :value ::c}
            ::c {:type :schema, :child {:type 'pos-int?}}}}
2️⃣ less noicy:
{:type :and,
 :children [{:type ::a}
            {:type ::b}
            {:type ::c}]
 :registry {::a {:type ::b}
            ::b {:type ::c}
            ::c {:type :schema
                 :child {:type 'pos-int?}}}}
3️⃣ compact:
{:type :and,
 :children [::a ::b ::c]
 :registry {::a ::b
            ::b ::c
            ::c {:type :schema
                 :child {:type 'pos-int?}}}}
#2021-09-2611:36Ben Sless3 looks best#2021-09-2613:02Karol Wójcik3#2021-09-2613:59ikitommiI think 3 is also the fastest, short-circuiting to lookup instead of walking more maps before the lookup. Pushed the registry into it's own key, it's also faster to use (one map lookup less) and looks better. So, ast is map with :type, optionally :properties and :registry. Plus any keys Schema wasn't to lift there. There are few comventions with helpers: • no child's -> nothing added • one child -> :child • many childs -> :children • same, but not schemas, just values (e.g. [:> 1], [:enum "small" "large"] -> :value and :values • entries -> :keys with maps with :value, optionally :prooerties and :order ... but, could have anything, like :=> has :input & :output#2021-09-2613:59ikitommiI think 3 is also the fastest, short-circuiting to lookup instead of walking more maps before the lookup. Pushed the registry into it's own key, it's also faster to use (one map lookup less) and looks better. So, ast is map with :type, optionally :properties and :registry. Plus any keys Schema wasn't to lift there. There are few comventions with helpers: • no child's -> nothing added • one child -> :child • many childs -> :children • same, but not schemas, just values (e.g. [:> 1], [:enum "small" "large"] -> :value and :values • entries -> :keys with maps with :value, optionally :prooerties and :order ... but, could have anything, like :=> has :input & :output#2021-09-2713:28jkrasnayHi, I have a schema that looks like this:
(def MySchema [:or
                 :string
                 [:and
                  [:vector any?]
                  [:fn {:error/fn
                        (fn [{:keys [value]} _]
                          (str "Not found: " (pr-str value)))}
                   my-lookup]]])
#2021-09-2713:29jkrasnayWhen I run (-> (m/explain MySchema 42) (me/humanize)) the result is ["should be a string" "invalid type" "Not found: 42"]#2021-09-2713:31jkrasnayWhat I’m hoping for is a way to organize the schema so that the error is something like “should be a string or a vector”, and if the value is a vector and my-lookup fails the error is simply “Not found: [:whatever]“. Is there a way to do this?#2021-09-2713:38jkrasnay(also it would be nice if my-lookup didn’t get called if the value wasn’t a vector. I suppose [:and] does not short-circuit)#2021-09-2713:53jkrasnayI suppose I could replace my-lookup with #(or (not (vector? %)) (my-lookup %)), then put an :error/fn on the :vector check that returns “should be a vector”.#2021-09-2714:04jkrasnayStill interested if there’s a more elegant way to do this, though.#2021-09-2714:33ikitommi@jkrasnay there is a hidden feature where you can add the error-properties into parent-nodes: in case a child errors, the too-most error is used instead, once for all errors. I was not happy with the impl and didn't need it in my client project, so decided not to release it.#2021-09-2714:36ikitommitry: https://github.com/metosin/malli/blob/master/test/malli/error_test.cljc#L499-L524#2021-09-2714:58jkrasnayOK, thanks. I’ll take some time to study these.#2021-09-2714:39ikitommi:and and :or short-circuit & collect all errors - it would be easy to inject extra data to those explanations and add options to either schema itself (via properties) or the me/humanize to allow control how to work with them.#2021-09-2717:25NicholasHello there, I've just started a very basic template with Shadow-cljs and Malli , but as I run npx shadow-cljs watch :app I get an error saying : *Wrong number of args (3) passed to malli.core/-fail!* . The project itself is just here : https://github.com/bitbot123/shadow-cljs-malli/blob/master/src/core.cljs (on the master branch) , and I've stuck [metosin/malli "0.6.1"] in shadow-cljs.edn , is this correct? Any help is much appreciated.#2021-09-2718:12ikitommioh well, released version with just the minimal fixes for that @shamansandtheprimes :
➜  ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.6.2"}}}'
Downloading: metosin/malli/0.6.2/malli-0.6.2.pom from clojars
#2021-09-2718:13ikitommihttps://github.com/metosin/malli/commit/98167660c841fdf9618974a83e7e7d2fa3acd5b2#2021-09-2811:30NicholasThe project compiles no problem, thank you @ikitommi#2021-09-3008:02Yehonathan SharvitMalli mentions JSON schema as a source of inspiration (in malli's https://github.com/metosin/malli#links-and-thanks). Here is an https://blog.klipse.tech/javascript/2021/09/30/data-validation-with-json-schema.html I wrote about JSON schema validation that mentions Malli. The loop is closed 😎#2021-10-0113:32ikitommiGood article. I think you could implement the Full JSON Schema with malli. Maybe Malli should be rebranded as an library to create schema libraries 👻#2021-10-0310:27Yehonathan SharvitWhat would it take to implement JSON Schema with malli @U055NJ5CC?#2021-10-0114:43zaneWhat is the meaning of the error :malli.core/potentially-recursive-seqex?#2021-10-0116:09ikitommiI believe it's documented on README &/ source code.#2021-10-0116:21zaneI looked through both and it’s not readily apparent to me.#2021-10-0116:34ikitommiI'll try: it's a design choice to keep the parsing fast and simpler: You can't recur from a sequence itself, but you can recur from a element in the sequence. http://malli.io has an example how hiccup works.#2021-10-0116:35ikitommie.g. wrap the :ref inside a :schema forcing it out of the secuence to it's own schema.#2021-10-0116:45zaneThat’s extremely helpful. Thanks!#2021-10-0120:21Carsten BehringI am struggling to define a schema as defn metadata, which accepts any values as inpu (as I only one to validate output) How could this look like ?#2021-10-0122:18respatializedassuming your function is 1-arity, you could do:
(defn my-func {:malli/schema [:=> [:cat :any] output-schema]}
  ... ; body goes here)
#2021-10-0214:53respatializedIs there an option for m/parse that disables returning tagged values for :orn / :altn / :multi / etc?#2021-10-0215:09ikitommicurrently, no. What is your use case for this?#2021-10-0215:10ikitommiif you want to disable all, just call validate and return the original value in case of success ..#2021-10-0215:26respatializedI have a large, complex schema that implements a https://github.com/fabricate-site/fabricate/blob/c04afeb0ab40b49f1f95bde126333992b25ae644/src/site/fabricate/prototype/html.clj#L150 for Hiccup elements. I leverage :orn throughout to preserve contextual information about elements (such as whether they're flow content or phrasing content). This is what it looks like when using a parser to parse a small element:
(site.fabricate.prototype.html/parse-element [:p "text" [:em "with emphasis"]])
=>
[:flow
 [:p
  {:tag :p,
   :attrs nil,
   :contents
   [[:atomic-element [:text "text"]]
    [:node
     [:em
      {:tag :em,
       :attrs nil,
       :contents [[:atomic-element [:text "with emphasis"]]]}]]]}]]
While this contextual information is sometimes useful, I also want the option of returning parsed values without it: just the {:tag t :attrs {} :contents [...]} structure so that every result from m/parse is returned in a uniform way. Perhaps, as you suggest, I'm relying too heavily on m/parse here; there's not much stopping me from pattern matching on the head & tail of the actual elements (so long as they validate) using ordinary sequence functions.
#2021-10-0215:28respatializedjust as an aside, the fact that it's possible and performant to implement this much contextual information using seqexes speaks to the expressive power of malli. 🔥#2021-10-0216:29respatializedafter rewriting the schema manually, I think I was confused about how tagged entries work. if tagging were disabled for the schema, then I wouldn't even get the map syntax from the result.#2021-10-0216:30respatializedmy intended result was a mixture of tagged and untagged parse results, so it wouldn't even have been achieved by disabling tagging for :orn.#2021-10-0312:41andre.richardsEdit: I implemented below by overriding malli.json-schema/-schema. In there I can control whether to return the ref (and add it to definitions) or return the deref of the attribute schema. Is there a way to 'deref' [:map] children when transforming a schema to Swagger? We use 'decomplected' map attributes, e.g.:
(def registry
    (merge
      (m/default-schemas)
      {:customer/id   :uuid
       :customer/name :string
       :Customer      [:map
                       [:customer/id]
                       [:customer/name]}))

(transform :Customer {:registry registry})
Results in:
{:type        "object",
 :properties  {:customer/id   {:$ref "#/definitions/:customer/id"},
               :customer/name {:$ref "#/definitions/:customer/name"}},
 :required    [:customer/id :customer/name],
 :definitions {:customer/id {:type "string", :format "uuid"}, 
               :customer/name {:type "string"}}}
Apart from the fact that this gives an error in Swagger (the extra forward slash after customer in $ref value), we would really like to get Swagger that looks like this (i.e. without a definition for each attribute):
{:type       "object",
 :properties {:customer/id {:type "string", :format "uuid"}, 
              :customer/name {:type "string"}},
 :required   [:customer/id :customer/name]}
We re-use the attributes in multiple schemas, and really like the fact that we don't have to re-define the attribute schema every time we use it (which is the point of decomplected attributes), and this is working as expected for all the core Malli operations like validate. We considered calling m/deref-all on the :Customer schema, but [:map] does not deref its children, so this does not help.
#2021-10-0411:52Yehonathan SharvitWhat is the relationships between decode and validate? Are we suppose to validate after a value has been successfully decoded?#2021-10-0411:57Ben SlessDecode never "fails", it's a best effort to get data to look like what should be validated. Validation should be done after decode#2021-10-0411:58Ben Slessafter decode and before validation you can't really know if the data is valid, yet#2021-10-0412:03Yehonathan SharvitMakes sense @UK0810AQ2 . Thanks#2021-10-0416:01Carsten BehringHow can I do a schema, which allows maps of "any key" of a certain type , string for example (or qualified symbols)#2021-10-0416:08Carsten Behringfound it, :map-of#2021-10-0423:23onetomis anyone aware of any work being done on OpenAPI (v2/v3) to Malli conversion? we are using https://github.com/oliyh/martian to talk to many APIs, and we love it, but it's using Plumatic Schema and we would like to use Malli instead.#2021-10-0423:53onetomim just watching https://www.youtube.com/watch?v=gMYQ1vDy7d0 and learnt that OpenAPI v3.1 is gonna be fully compatible with JSON Schema draft 8#2021-10-0509:54Carsten BehringI could not get my head arround the best practices to use Malli for runtime checking of function parameters. I saw the docu about this, but wonder between "instrumentation", which seems to be dev oriented,#2021-10-0509:55Carsten Behringvs call validate in the code#2021-10-0509:55Carsten BehringI store my schema as function metadata if this helps.#2021-10-0509:56Carsten BehringHow do people do this ?#2021-10-0514:40danielglauserWhen working with Malli we used it to test all API inputs to the system and sometimes the outputs from the system. We didn't use it to check common function calls. Why do you want to do that?#2021-10-0514:50Noah Bogartfor development it’s very nice to see when you meant to pass in a vec and passed in a map instead, but in prod i feel like those kinds of checks aren’t helpful (especially if the code actually runs how you expect anyways)#2021-10-0515:15danielglauserI can see that though I usually accomplish those types of checks through my tests. It'll be interesting to see in anyone has a technique for using Malli for environment based assertions.#2021-10-0515:21ikitommi@danielglauser it’s layered: malli.instrument can be used in prod too, it doesn’t contain any var-watches, clj-kondo emitting and thanks to it’s filters, you can annotate your functions with meta like {:malli/always-validate true} and instrument those in prod.#2021-10-0515:23ikitommihave worked with a lot of codebases with Plumatic Schema where few (important) functions had the :always-validate on. e.g. when reading data from document-db or just Very Important Function (handling money) where the perf penalty is ok.#2021-10-0515:25ikitommimalli.dev is built on top of malli.instrumentwith just start! and stop! , which both are stateful. would not use in prod.#2021-10-0515:27ikitommistateful as in watching the Vars and emitting (clj-kondo) files.#2021-10-0522:29Carsten BehringI think "input checking" at runtime can be important for public functions. At the end, I only guarantee that my function works, if the caller does his part (giving valid input only). Failing so, resulst often in very ugly error message deep in Clojure, from which it is impossible to read what I have done wrong (as teh caller).#2021-10-0522:30Carsten Behring"Output" checking is a complete different story. That I probably don't want to have in production code.#2021-10-0522:31Carsten BehringIs somebody doing this ? I see various way to implement the parameter checking, and was wondering which works best. 1. Instrumenting #2021-10-0522:31Carsten Behring2. direct calls of format#2021-10-0522:32Carsten BehringTogether with this come the different options, where to keep the schema ? as defn metadata or in some vars ? How can we the use those easely for documentation of a function ?#2021-10-0619:32agigaoHey guys, any pointers how can I check/validate a value for bigdec?#2021-10-0619:42ribelohttps://clojuredocs.org/clojure.core/decimal_q#2021-10-0619:44agigaoOh, thanks!#2021-10-0807:19HukkaWe didn't have a test for it, and managed to (of course) put JSON schema example data that didn't match the schema, so the swagger ui happily offered to make requests that gave an error. I tried to look for it, but couldn't find any: is there a ready made helper that creates a sample data structure from the embedded :json-schema/examples, or should we just walk through the schema tree and built it ourselves?#2021-10-1316:36Carsten Behringcan I easely declare in a schema that all keys are optional ?#2021-10-1318:19HukkaEasily as data, no, but there are helpers that transform a schema recursively to open or closed#2021-10-1318:19HukkaWhoops, what am I saying, that'd completely different, of course#2021-10-1318:20HukkaThere's a separate helper for opening and closing#2021-10-1318:21HukkaAnd then others for optional and required#2021-10-1319:03Carsten Behringok, thanks#2021-10-1322:30lsenjovmalli.util has basically everything you’ll need#2021-10-1322:33lsenjovSpecifically, optional-keys#2021-10-1322:33lsenjovThere’s also malli.core/children for getting all keys out#2021-10-1411:45Ben SlessI'm trying the instrument example from the documentation, while I get the report in the REPL, kondo isn't emitting the lint warnings for it. Am I missing anything?#2021-10-1411:46borkdude@ben.sless Have you added the emitted config to your clj-kondo config?#2021-10-1411:48Ben SlessLooks like I am missing something. The emitted config is written by malli by default to .clj-kondo/configs/malli/config.edn, do I need to include it somehow?#2021-10-1411:49borkdudeyeah, in .clj-kondo/config.edn you need to write {:config-paths ["configs/malli"]}#2021-10-1411:50Ben SlessWoohoo!#2021-10-1411:51Ben SlessThank you 🙂#2021-10-1813:15andre.richardsThanks, this solved it for me as well 🙏 A question on clj-kondo/malli integration: when a function takes a map as parameter, clj-kondo is highlighting if the passed parameter is not a map, but it does not highlight if the value under a certain key is not correct type. Is this expected, and a hard limitation of clj-kondo/malli, or something that just has not been done (because it will require a lot of work, etc.). I just want to know where to set my expection. 😁#2021-10-1412:02jussiHey, I ran into the following when converting a schema to PlantUML.
(def value
  [:and
   {:description "Non negative numeric value."
    :json-schema/example 780.56}
   number?
   [:>= 0]])

(def simple-registry
  (merge
   (m/default-schemas)
   (mu/schemas)
   {:value value}))

(def schema-merged
  (mu/closed-schema
   (m/schema
    [:merge
     [:map 
      [:name string?]
      [:age value]]
     [:map
      [:address string?]]]
    {:registry simple-registry})))

(comment
  (plantuml/transform schema-merged))
Now, trying to render PlantUML results in :malli.core/invalid-schema {:schema :merge}. Does plantuml/transform support stuff outside malli.core? The schema-merged above expands into following schema:
[:merge [:map {:closed true} [:name string?] [:age [:and {:description "Non negative numeric value.", :json-schema/example 780.56} number? [:>= 0]]]] [:map {:closed true} [:address string?]]]
#2021-10-1419:51dvingoNot sure why the plantuml transform doesn't do this but if you deref the schema, it works:
(plantuml/transform (m/deref schema-merged))
#2021-10-1421:10jussiThanks! Missed that completely 🙈#2021-10-1421:29Ivan FedorovWhat’s the replacement for a set as a predicate in malli? Analogue of (s/def :spec1 #{:a :b}) Is it [:or [:= :a] [:= :b]] ?#2021-10-1422:26schmee[:enum :a :b] :thumbsup:#2021-10-1422:50Ivan Fedorovright! danks @U3L6TFEJF#2021-10-1615:47chaosHi, is it possible to restrict the range of values for integer predicates produced by the generator ? For example (malli.generator/generate [:int {:min 0 :max 6}]) reduces the range to [0,6], but (malli.generator/generate [int? {:min 0 :max 6}]) , where int? is an integer predicate, does not. My actual use case is with pos-int?. Thanks#2021-10-1615:52Ben Slesspos-int? is equivalent to [:int {:min 1}] Type schemas are more flexible than predicate schemas#2021-10-1615:53Ben SlessYou can then add upper boundary with no complications#2021-10-1616:04chaosExcellent thanks! This is what I was looking for [:int {:min 0 :gen/max 10}] ; i.e. eq to nat-int? for validation and [0,10] for the generated range#2021-10-1616:17Ben Slessyep#2021-10-1616:17Ben SlessWhere types are concerned, I prefer the type schemas (with keywords) and not the predicate schemas#2021-10-1616:18Ben SlessYou use case is a perfect example why#2021-10-1615:50Ben SlessWhen transforming multi to json schema, how about using implications? https://json-schema.org/understanding-json-schema/reference/conditionals.html#id7
#2021-10-1620:59chaosIs there a generator option to produce distinct values for a vector? for example the following schema generated some values more than once
(->> [:vector {:min 400} string?]
       malli.generator/generate 
       frequencies
       (filter (fn [[_ v]] (> v 1))))
;; => (["" 20] ["e" 2] ["7" 2])
Thanks
#2021-10-1703:45Ben SlessUse a set generator then fmap vec#2021-10-1707:42chaosGreat thanks! Although this has been a little bit of a mouthful it has done the trick:
[:vector {:min 400 
          :gen/schema [:set {:min 10} string?]
          :gen/fmap vec} 
         string?]
#2021-10-1707:49chaos(perhaps there should be a :gen/distinct option for collections? clojure.test.check.generators does seem to provide list-distinct and`vector-distinct` to this end)#2021-10-1721:01chaosThis unfortunately does not scale very well, because I'd need to repeat the vector predicate twice, one for :vector the other for the generator's schema, which could easily get out of sync for more complicated schemas, e.g.
[:vector {:min 400
          :gen/schema [:set {:min 10} [:map [:x int?] [:y string?]]]
          :gen/fmap vec
         }
         [:map [:x int?] [:y string?]]]
#2021-10-1803:45Ben SlessAt that point you might want to define your own type and use a registry, you could call it distinct vector. Why does it need to be a vector, btw? Why can't it be. a set?#2021-10-1817:00chaosHi, it has to be a vector because this is how values are coming out from the data source. The distinct values are only needed by the generator to produce some specific values during testing only. I can't seem to create a custom type with -simple-schema for distinct-vector ... The syntax of the type looks to be in accordance with the section on the readme file, but it gives me a :malli.core/invalid-schema {:schema (#object[cljs$core$string_QMARK_])} error when I try to use it ...
(let [vector-distinct
      (malli.core/-simple-schema
        (fn [_ schema]
          (let [schema (first schema)])
          {:type :vector-distinct
           :pred vector?
           :type-properties
           {:gen/gen (clojure.test.check.generators/vector-distinct (malli.generator/generator schema))}}))]
  (-> [vector-distinct {} string?]
      malli.generator/generate))
Thanks!
#2021-10-1818:17Ben SlessI think you want a variation on vector-of#2021-10-1818:17Ben Slessthe schema itself is incorrect, not the generation#2021-10-1819:15chaossorry, not sure what you mean by "a variation on vector-of"; but I think I got it this time working, It appears I was missing the number of arguments accepted by the new type, in my case 1 (as in :min and :max):
(let [vector-distinct
        (malli.core/-simple-schema
         (fn [_ schema]
           (let [schema (first schema)]
             {:type ::vector-distinct
              :min 1
              :max 1
              :pred vector?
              :type-properties
              {:gen/gen (clojure.test.check.generators/vector-distinct (malli.generator/generator schema))}})))]
    (->> [vector-distinct string?]
        malli.generator/generate))
Does this look sane or is it just too hacky?
#2021-10-1906:19chaosActually that is not complete because pred is not going to check the vector values for conformity. I will stick with the following hack which almost does what I want (the only glitch is that generated distinct values are likely to be less than :min):
[:vector {:min 400 :max 400
          :gen/fmap #(into [] (set %))} string?]
#2021-10-1907:02Ben Slessah, you need a collection schema, that's it#2021-10-1814:16andre.richardsThe malli pretty printer (virhe) is failing when it tries to print a datomic db. It might not be a defect in malli/virhe (see details below), so is there a way to exclude certain fields from pretty printing as a workaround? I'm starting malli.dev with
(malli.dev/start! {:report (pretty/reporter)})
Take for example this function:
(defn fetch-user
  "Fetches a user by its entity `id`. If no entity is found, nil is returned."

  {:malli/schema
   [:=> [:cat :int :any]
    [:maybe :map]]}

  [id db]

  ...)
The db (second) parameter is defined as :any - I'm not even interested in checking that it is a valid db right now, I just want to ensure the id is of valid type. When I call the function with a valid id (:int) and db: (fetch-user 96757023244461 db), it works as expected. However when calling it with a string or anything else in the first position (`id`), it results in this:
(fetch-user "96757023244461" (db/current))

Execution error at malli.dev.virhe.EdnPrinter/visit_set (virhe.cljc:80).
Unable to convert: class datomic.core.btset.BTSet to Object[]
i.e. it is not printing the malli error as it should, but instead it fails with the above. Passing anything else in the second argument, and still the (incorrect type) string in first argument, prints the malli error as expected:
-- Schema Error ---------------------------------------------------

Invalid function arguments:

  ["96757023244461" nil]

Input Schema:

  [:cat :int :any]

Errors:

  {:in [0],
   :message "should be an integer",
   :path [0],
   :schema :int,
   :type nil,
   :value "96757023244461"}
I have attached the stacktrace for when the error can not be pretty printed (i.e. when an actual datomic db is passed). Presumably it is traversing the db object's fields(?) and fails when it gets to the datomic.core.btset.BTSet and tries to convert it to an array. It's not clear to me why datomic's BTSet is failing at that point, and it might not be a defect in malli, but is there a way to exclude certain fields from pretty printing, in order to work around this issue?
#2021-10-1815:57ikitommioh, [malli.dev.virhe.EdnPrinter visit_set "virhe.cljc" 80] <- the datomic.core.btset.BTSet is kinda like set, but not?#2021-10-1816:01andre.richardsThat's what it looks like, yeah#2021-10-1815:58ikitommi
(visit-set [this x]
    (let [xs (sort-by identity (fn [a b] (arrangement.core/rank a b)) x)]
      (fipp.edn/pretty-coll this "#{" xs :line "}" fipp.visit/visit)))
#2021-10-1815:59ikitommi@andre.richards if you can make that sort-by wotk with datomic.core.btset.BTSet without extra deps, PR most welcome#2021-10-1816:02andre.richardsI will see what I can come up with#2021-10-1816:03andre.richardsWill maybe ask on Datomic forum what is up with that BTSet#2021-10-1817:42dominicmCalling seq on your argument might work.#2021-11-0111:52andre.richardsThis works. I will open an issue and a pull request.#2021-10-2009:31ikitommihi all. there is bunch of quality PRs being held in the queue, sorry. I’m working on with the malli internals (schema ast, lifecycle, providers, registries) and want to find a good design before merging anything big in. Will review and pull the queued PRs after that. Any small fixes welcome anytime.#2021-10-2009:32Ben Sless👍#2021-10-2009:32Ben SlessNecroing a question I asked some time ago, any idea for a transformer which removes invalid keys?#2021-10-2009:33Ben Sless(given that they're optional)#2021-10-2009:34ikitommiinvalid… you could a) validate the keys when transforming or b) explain and remove based on that.#2021-10-2009:46Ben SlessI think I can build (a), the transformer has access to the parent schema#2021-10-2009:47Ben SlessAlso reminds me, I think an interleaved transformer/validator will be good#2021-10-2009:48ikitommidoing both in a single pipeline?#2021-10-2009:49Ben Slessyes#2021-10-2009:49Ben SlessThe decoder can end up doing lots of allocations#2021-10-2009:49Ben Slessyou can short circuit on it#2021-10-2009:50ikitommibut, you can’t validate before it’s transformed, right?#2021-10-2009:50Ben Slessright#2021-10-2009:50ikitommiso, it would need to happen on :leave -> it’s all transformed already?#2021-10-2009:50Ben Slessyou'll have something like a coercer which transforms and validates in one pass#2021-10-2009:50Ben Slessthink so#2021-10-2009:51ikitommihow would it allocate less if that happens after the transformation?#2021-10-2009:51ikitommiwouldn’t all the nested childs get re-validated when you are leaving them?#2021-10-2009:52ikitommi
[:map [:a [:map [:b [:map [:c [:map [:d :boolean]]]]]
#2021-10-2009:53ikitommiunless the walking knows which childs are already transformed & validated.#2021-10-2009:54Ben Slesshm, generally, you have no way of knowing if you need to re-walk the children#2021-10-2009:54Ben Slessespecially if you do interesting transformations#2021-10-2009:56ikitommimy assumption is that having validation and transformation as separate steps is the fastest way to do it. 2 simple sweeps instead of one (more complex) sweep.#2021-10-2009:57ikitommibut, all ears if there is a better way.#2021-10-2009:59ikitommioriginally, i though of doing all the workers using just -walk. there would be walker to create a validator, decoder etc. but as all schemas should have all of those and as performance was a one of the primary goals - they got separate (protocol) methods.#2021-10-2010:00ikitommithe one walker would have allowed to compose a chain of validate + transform in an easier way, I think.#2021-10-2010:04Ben SlessFigured out how to strip invalid optional keys, feel free to add it to tips, after some beautifying
(defn strip-invalid-optional-keys-transformer
  ([]
   (let [transform
         {:compile
          (fn [schema _]
            (let [entries (filter #(:optional (m/properties (second %))) (m/entries schema))
                  fs (map (fn [[k v]]
                            (let [validator (m/validator v)]
                              (fn [m]
                                (if-let [e' (find m k)]
                                  (let [v' (val e')]
                                    (if (validator v')
                                      m
                                      (dissoc m k)))
                                  m))))
                          entries)]
              (reduce comp fs)))}]
     (mt/transformer
      {:decoders {:map transform}
       :encoders {:map transform}}))))
#2021-10-2010:04Ben Sless
(m/decode [:map [:a {:optional true} int?] [:b {:optional true} int?]] {:a 1 :b 2.2} strip-invalid-optional-keys-transformer)
#2021-10-2010:05ikitommi👍#2021-10-2009:34ikitommia) might be cleaner (and faster)?#2021-10-2009:36ikitommi:or does also validation on transformation, as it needs to find the branch, which is valid after transformation.#2021-10-2009:38ikitommispike about caching computations (`-form`, -validator etc) with schemas:
(def schema
  (m/schema
   [:map
    [:x boolean?]
    [:y {:optional true} int?]
    [:z [:map
         [:x boolean?]
         [:y {:optional true} int?]]]]))

;; 1.5µs -> 11ns (130x)
(p/bench
 (m/validator schema)

;; 1.7µs -> 64ns (25x)
(p/bench
 (m/validate schema {:x true, :z {:x true}})) ; => true
#2021-10-2009:41ikitommithere is the initial cost of creating the thing, but just once opposed to every call. the results are cached with the actual schema instance, so when the schema instance is not needed, the cached results will also be GCd, so not leaking memory.#2021-10-2009:41ikitommiusing computation directly is still fastest, but not much:
;; 55ns
(let [validate (m/validator schema)]
  (p/bench
   (validate {:x true, :z {:x true}})))
#2021-10-2012:27Ben SlessPlease disregard last message, user error#2021-10-2115:53Ivan FedorovHeyy, do you already have hiccup / reagent form generation on malli?#2021-10-2118:59escherizeHi Ivan, depending on what you mean, I took an experimental crack at something like that a while back. it’s really buggy but you can see it live here: https://escherize.com/w/data-desk/#2021-10-2119:17Ivan Fedorovoh niccceee!#2021-10-2119:18Ivan Fedorovopen source?#2021-10-2119:23Ivan FedorovFound it! thanks a lot!#2021-10-2209:50ikitommilooks great!#2021-10-2118:59escherize
[:map [:a-single-int int?]
      [:a-single-string string?]
      [:a-single-bool boolean?]
      [:int-vekdor [:vector int?]]
      [:string-vekdor [:vector string?]]
      [:bool-vekdor [:vector boolean?]]
      [:map-vekdor [:vector [:map [:a-single-int int?]
                                  [:a-single-string string?]
                                  [:a-single-bool boolean?]
                                  [:int-vekdor [:vector int?]]
                                  [:string-vekdor [:vector string?]]
                                  [:bool-vekdor [:vector boolean?]]]]]]
#2021-10-2118:59escherizeif you paste that into the box on top, on blur the page should change into a ui to generate data that matches that schema#2021-10-2120:18Andrés RodríguezIs there a transformer to modify a single specific value based on its key path?#2021-10-2209:50ikitommi@hello525 sure, you could 1. put a transformer into top-level schema using schema properties as do update-in to the value 2. m/walk the schema and add the paths as schema properties to each schema. With this + :compile in transformer, you can select the schemas that need to changed at transformer creation time. There is malli.util.subschemas which could be looked as example imp how to find paths for each (nested) schema element#2021-10-2211:56martinklepschIs there a function like validate that will throw instead of returning a value?#2021-10-2212:15ikitommiat the moment, no. If that would he added, there would be a reason to add throwing validator, explain, explainer, parse, parser, transform, transformer too#2021-10-2213:34martinklepschCool, it’s fine really, just wanted to check 🙂#2021-10-2214:06Karol WójcikHowever you can create such function yourself 😄#2021-10-2214:07Karol WójcikSomething like:
(defn- explain!
     [aschema x fn-name]
     (when-let [info (m/explain aschema x)]
       (when (or cfg/BROWSER? cfg/TEST_ENV?)
         (throw (ex-info "[Malli] Schema does not match the argument"
                         {:type :malli-error
                          :x (form->pprint-str x 3)
                          :fn-name fn-name
                          :info (form->pprint-str (me/humanize info) 3)
                          :schema-name (schema-name aschema)
                          :schema-doc (schema-doc aschema)
                          :schema (schema->form-pprint-str (schema-form aschema))})))))
#2021-10-2214:34ikitommisome perf numbers 0.6.1 vs 0.7.0 (unreleased):
(def ?schema
  [:map
   [:x boolean?]
   [:y {:optional true} int?]
   [:z [:map
        [:x boolean?]
        [:y {:optional true} int?]]]])

;; 44µs -> 2.9µs (15x)
(p/bench (m/schema ?schema))

;; 240ns (180x, parses entries when actually needed)
(p/bench (m/schema ?schema {::m/lazy-entries true}))

(def schema (m/schema ?schema))

;; 1.7µs -> 64ns (25x)
(p/bench (m/validate schema {:x true, :z {:x true}}))
#2021-10-2509:21Ben Slesshow should I compile a transformer which I want to operate only on leave?#2021-10-2511:27Ben SlessIt's pretty terrible but it works. Anything here a bad idea?
(defn -strip-invalid-optional-keys-transformer
  [schema _]
  (let [entries (filter #(:optional (m/properties (second %))) (m/entries schema))
        fs (map (fn [[k v]]
                  (let [validator (m/validator v)]
                    (fn [m]
                      (if-let [e' (find m k)]
                        (let [v' (val e')]
                          (if (validator v')
                            m
                            (dissoc m k)))
                        m))))
                entries)]
    (reduce comp fs)))

(def strip-invalid-optional-keys-transformer
  (let [transform {:compile
                   (fn [schema _]
                     {:leave -strip-invalid-optional-keys-transformer})}
        encoders {:map transform}]
    (mt/transformer
     {:decoders encoders
      :encoders encoders})))
#2021-10-2516:51ikitommiwith a quick look, looks good. the runtime is bare minimum, everything possible has been pushed to creation time.#2021-10-2516:51ikitommibut, should the encoding happen at enter?#2021-10-2516:52ikitommie.g. you have valid value, after encoding, the keys could be different, (e.g. stringified) so the transforming functions misses all the keys.#2021-10-2518:49Ben Slessyou're correct, I didn't consider the encode case#2021-10-2518:50Ben SlessI must say the order where things happen with regards to transformers compilation was very hard to follow 😞#2021-10-2511:27Ben SlessIt's pretty terrible but it works. Anything here a bad idea?
(defn -strip-invalid-optional-keys-transformer
  [schema _]
  (let [entries (filter #(:optional (m/properties (second %))) (m/entries schema))
        fs (map (fn [[k v]]
                  (let [validator (m/validator v)]
                    (fn [m]
                      (if-let [e' (find m k)]
                        (let [v' (val e')]
                          (if (validator v')
                            m
                            (dissoc m k)))
                        m))))
                entries)]
    (reduce comp fs)))

(def strip-invalid-optional-keys-transformer
  (let [transform {:compile
                   (fn [schema _]
                     {:leave -strip-invalid-optional-keys-transformer})}
        encoders {:map transform}]
    (mt/transformer
     {:decoders encoders
      :encoders encoders})))
#2021-10-2515:37Ben SlessIt seems that most times where relations about map keys are requires by users it is to specify mutual exclusion. Could it be worth it to add a mutex property which takes a collection of keys which are mutually exclusive?#2021-10-2516:46ikitommiwoudn’t the key-relations solve that? havan’t had time to give that much though, but agree something should be there in the core.#2021-10-2518:49Ben SlessGetting the key-relations to work correctly and with good performance will be lots of work which I don't know when I'll get to. Adding :mutex [[:a :b]] can be done in a couple of hours#2021-10-2616:12Ben SlessFYI regarding JSON schema, might be worth to chime in https://github.com/json-schema-org/community/discussions/70#2021-10-2712:48ikitommilooks interesting#2021-10-2712:48ikitommithanks for sharing.#2021-10-2712:51Ben SlessMight be a good idea to join their slack, too. Maybe spec and malli can give them ideas#2021-10-2712:51ikitommi👍#2021-10-2712:44Ivan FedorovHas anyone seen malli to datomic schema translation projects?#2021-10-2713:31winsomeThere was a pretty good talk about using malli for this from summer 2020, I think.#2021-10-2713:31winsomehttps://www.youtube.com/watch?v=ww9yR_rbgQs#2021-10-2713:31winsomeIt's not just malli->datomic, but IIRC there's enough information here to figure it out.#2021-10-2713:35Ivan Fedorov@U028BUU1P3R talk is awesome, thanks! Is there any underlying open source?#2021-10-2713:36Ivan FedorovI have the experience and the will to write such generator, a question is if I’ll be duplicating someone’s efforts#2021-10-2713:53winsomeI don't think there is any source shared from this talk, but I was able to get something similar working myself but with datahike. I ran into some trouble getting the described "ref" system working, but even so I was able to get a pretty decent result.#2021-10-2717:47pithylessNot sure how much is covered, but there has been some work done for eql and specs: https://github.com/dvingo/malli-code-gen#2021-10-3118:06ikitommiI guess everyone is bored on the benchmarks, but updating docs, added plumatic schema to the benchmark comparison. The code looks legit :thinking_face:
(def data {:x "true", :y "1", :z "kikka"})
(def expexted {:x true, :y 1, :z "kikka"})

(spec/def ::x boolean?)
(spec/def ::y int?)
(spec/def ::z string?)

;; clojure.spec 19µs
(let [spec (spec/keys :req-un [::x ::z] :opt-un [::y])
      transform #(st/coerce spec % st/string-transformer)]
  (assert (= expexted (transform data)))
  (cc/quick-bench (transform data)))

;; plumatic schema 2.2µs
(let [schema {:x schema/Bool
              (schema/optional-key :y) schema/Int
              :z schema/Str}
      transform (sc/coercer schema sc/string-coercion-matcher)]
  (assert (= expexted (transform data)))
  (cc/quick-bench (transform data)))

;; idiomatic clojure 290ns
(let [transform (fn [{:keys [x y] :as m}]
                  (cond-> m
                    (string? x) (update :x #(Boolean/parseBoolean %))
                    (string? y) (update :y #(Long/parseLong %))))]
  (assert (= expexted (transform data)))
  (cc/quick-bench (transform data)))

;; malli 72ns
(let [schema [:map
              [:x :boolean]
              [:y {:optional true} int?]
              [:z string?]]
      transform (m/decoder schema (mt/string-transformer))]
  (assert (= expexted (transform data)))
  (cc/quick-bench (transform data)))
#2021-10-3119:08Ben Slessnow just waiting for the protocols to move namespace and time schemas 🙂#2021-10-3118:08ikitommibtw, got the AST registry references working, could push 0.7.0 as soon as have verified it works on real projects. Then, away from perf, back to features.#2021-10-3119:30Ben Slessbtw, I have a pretty extensive getting started guide which is probably ready for initial contribution. Which format would you like it in? adoc/md/org?#2021-10-3121:36ikitommilooking forward to this! If the docs would be part of the repo, then adoc / md, so that github & cljdoc can render these. I have wanted to move to adoc, but hadn’t had time to learn the syntax differences….#2021-11-0104:14Ben SlessI'll just use pandoc to convert It's supposed to be a superset of md#2021-10-3121:41ikitommi[metosin/malli "0.7.0-20211031.202317-3"] is out, perf + new Schema AST. Will run it against work projects next week before making a real release. Has breaking changes, mostly in the extender API, so please read the https://github.com/metosin/malli/blob/master/CHANGELOG.md#070-20211031202317-3-2021-10-31.#2021-11-0116:58Felipe Cortezhi! the doc says m/=> can be placed both before or after a defn. when putting it before, clj-kondo complains about the symbol being unresolved. :unresolved-symbol {:exclude [plus]} in clj-kondo's config.edn silences the error. should that be the default behavior?#2021-11-0116:59borkdudethat's usually not an optimal solution to unresolved symbols in clj-kondo#2021-11-0116:59borkdudedoes malli export hooks and/or config as part of the library?#2021-11-0117:00Felipe Cortezit seems to export the stuff that goes to .clj-kondo/configs/malli/config.edn automatically#2021-11-0117:03borkdudeI don't think malli exports any macro config. Does it have macros that clj-kondo doesn't understand?#2021-11-0117:05Felipe Cortezthe only thing it seems to export for m/=> is
{:linters
 {:type-mismatch {:namespaces {brincando {plus {:arities {2 {:args [:int :int], :ret :int}}}}}}}}
but since the macro would still be out of order even if defined for clj-kondo, is there any other way to solve this?
#2021-11-0117:06borkdudePlease give an example, I find it hard to imagine what you're talking about#2021-11-0117:07Felipe Cortezsorry! this is the working code
(m/=> plus [:=> [:cat [:int {:max 10}] :int] :int])
(defn plus [x y] (+ x y))
but I just noticed that if you "expand" that macro in clj-kondo to declare plus, it solves the undeclared variable warning, right?
#2021-11-0117:08borkdudeI think you can just put the m/=> expression after the defn and then clj-kondo will understand that plus is a var#2021-11-0117:09Felipe Cortezyep, but you can also put m/=> before the defn and it is supposed to work, and I'd like to have the option to put the function schema close to the top of the defn#2021-11-0117:09borkdudeI understand#2021-11-0117:10borkdudeIn this case it's better to write a hook. Optimally malli itself would bundle this in the library and export it#2021-11-0117:10Felipe Cortezcool!#2021-11-0117:10borkdudealternatively you can use {:linters {:unresolved-symbol {:exclude [(malli.core/=>)]}}}#2021-11-0117:11borkdudethis will suppress all unresolved symbols in =>#2021-11-0117:11borkdudebut that config could also be exported by malli so it would work for everyone. cc @ikitommi#2021-11-0219:49ikitommiso, definetely. so, whoever knows better what to put into malli repo so it would work, please do.#2021-11-0219:49ikitommioh, that, I can do it#2021-11-0220:05ikitommi@UKW2FUL4D https://github.com/metosin/malli/pull/559. thing is, that if that is merged, clj-kondo will not warn on cases where you define the m/=>, but not the actual function that it points to. Not good either.#2021-11-0220:06ikitommibut, that’s stops the warnings.#2021-11-0220:56borkdude@ikitommi That's a step in the right direction, but you can make it better using an exported macro hook#2021-11-0220:56borkdudebut this is more time consuming and I understand the trade-offs#2021-11-0220:57borkdudeit would be better to export it as part of the malli library though, then people don't have to run malli first to get the better linting#2021-11-0221:06ikitommia better solution / clj-kondo macro hook would be great, anyone? Just merged the initial fix, thanks @borkdude for the code to paste in 🙂#2021-11-0313:51Felipe Corteznice! what about a hook that interprets m/=> as (fn [sym _schema] (clojure.core/declare %1))? kondo doesn't seem to complain about
(def thing)
(declare thing)
so it works the other way around too
#2021-11-0317:05borkdudeI would say:
(do (declare thing) schema)
so everything you use in the schema is still seen by clj-kondo.
#2021-11-0412:44Feliperight! I'll try this 🙂#2021-11-0623:40Felipe@ikitommi hmmm, apparently doesn't seem to catch errors for instrumented functions doing m/=> before the defn:
(require '[malli.core :as m])
  (require '[malli.dev :as dev])
  (dev/start!)

  (defn plus1 [x] (inc x))
  (m/=> plus1 [:=> [:cat :int] [:int {:max 6}]])
  (plus1 6)
  ;; 1. Unhandled clojure.lang.ExceptionInfo
  ;;  :malli.core/invalid-output {:output [:int {:max 6}], :value 7,
  ;;  :args [6], :schema [:=> [:cat :int] [:int {:max 6}]]}

  (m/=> plus1' [:=> [:cat :int] [:int {:max 6}]])
  (defn plus1' [x] (inc x))
  (plus1' 6)
  ;; => 7
#2021-11-0710:28ikitommi@UA2U3KW0L that’s not good. I guess (re-)calling (dev/start!) after definitions would work.#2021-11-0710:29ikitommiso, that’s just dev-time annoyance, but then again, it’s supposed to be dev-time tooling. Ideas welcome#2021-11-0710:30ikitommiI would assume m/=> with dev running, should put a var-watcher that takes care of that, but guess not :thinking_face:#2021-11-0712:22Felipeit seems to put a watch on the function schema, so what seems to be happening is the watch fn is called, the function gets instrumented properly, but when you eval the defn again you lose the instrumentation#2021-11-0712:25Felipeyep! as a quick test, I did
(defn start!
   ,,,
   (let [watch (fn [_ _ old new]
                 (println "watched")
                 (future ;; <-
                   (Thread/sleep 500) ;; <-
                   (mi/instrument! ,,,))]
     (add-watch @#'m/-function-schemas* ::watch watch))
   (mi/instrument! ,,,)
   ,,,))
#2021-11-0713:03ikitommiso, we should get an event when the var is redefined, e.g. via defn. I guess that's doable?#2021-11-0714:23Felipe Cortezadd-watch supports this, apparently:
(def changes* (atom []))  
  (def a 1)
  (add-watch #'a nil (fn [_ _ & old+new] (swap! changes* conj old+new)))
  (def a 2)
  (def a 3)
  @changes* ;; => [(1 2) (2 3)]
#2021-11-0714:23Felipe Cortezjust noticed I use clojurians through 2 different slack accounts#2021-11-0117:08borkdudeor you can use {:lint-as {malli.core/=> clojure.core/def}}#2021-11-0117:08borkdudewhich is not entirely semantically correct, but I think it works for linting#2021-11-0117:09borkdudehmm#2021-11-0117:09borkdudeno#2021-11-0117:09borkdudethen you will get a redefined warning :)#2021-11-0117:09Felipe Cortezclojure.core/declare, maybe?#2021-11-0221:06ikitommia better solution / clj-kondo macro hook would be great, anyone? Just merged the initial fix, thanks @borkdude for the code to paste in 🙂#2021-11-0623:40Felipe@ikitommi hmmm, apparently doesn't seem to catch errors for instrumented functions doing m/=> before the defn:
(require '[malli.core :as m])
  (require '[malli.dev :as dev])
  (dev/start!)

  (defn plus1 [x] (inc x))
  (m/=> plus1 [:=> [:cat :int] [:int {:max 6}]])
  (plus1 6)
  ;; 1. Unhandled clojure.lang.ExceptionInfo
  ;;  :malli.core/invalid-output {:output [:int {:max 6}], :value 7,
  ;;  :args [6], :schema [:=> [:cat :int] [:int {:max 6}]]}

  (m/=> plus1' [:=> [:cat :int] [:int {:max 6}]])
  (defn plus1' [x] (inc x))
  (plus1' 6)
  ;; => 7
#2021-11-0216:16dakraI have a beginner question: I validate incoming JSONs by first stripping extra keys, then validating against a malli schema and then output avro. How can I keep/specify the type of floats/doubles? E.g.
(-> [:map [:score double?]]
      (m/encode {:score 0} mt/strip-extra-keys-transformer)
      :score
      type)
;; => java.lang.Long
While I want double.
#2021-11-0216:21Ben SlessShould you use a string transformers first?#2021-11-0216:24dakraThis seems to work when it's a string. So
(-> [:map [:score double?]]
      (m/decode {:score "0"} mt/string-transformer)
      :score
      type)
;; => java.lang.Double
works.. but I get {:score 0} as input and then it's still Long.
#2021-11-0216:33dakraBut this lead me to look at the other transformers and decode with json-transformer seems to do the trick 🙂#2021-11-0216:37Ben SlessHang on, decode is when reading, encode is when writing#2021-11-0216:38Ben SlessIf the json is incoming you should decode#2021-11-0216:43dakraApparently I got that mixed up. But when I only use strip-extra-keys-transformer I can use encode or decode.. In both cases the extra keys got stripped. But for my (new) use-case where I want to have a double where the schema is double I have to use decode like you said.#2021-11-0217:02Ben SlessYes, where are you getting data in from? Http?#2021-11-0217:09dakraKafka#2021-11-0217:10dakraI write with jackdaw a kstreams app that validates the incoming JSON events, does some simple transformations and outputs to multiple different topics depending on the event content.#2021-11-0222:57Paul Santa ClaraHey there. I was just wondering if there were plans to expand malli to include support for openapi 3? Or should i just stick to spec-tools for the immediate future?#2021-11-0306:37HukkaThere's an issue for that where somebody was going to work on it, but I haven't got a reply last month. That said, I heard a rumour that someone else would tackle it by Christmas. Might be that I need it sooner and might have a look too. Didn't seem terribly difficult, considering that the jsonschema support is already there. At least a simple version that is valid openapi3. Perhaps something that handles all the refs etc. would be more complex.#2021-11-0313:04Paul Santa Clarathat's great to hear that it's tentatively on the roadmap! I'll keep a close eye on things and switch over ASAP. Thanks!#2021-11-0420:31respatializedIs there a recommended way to combine parsing and decoding/transforming operations? I'm trying to parse a value and return the transformed value (there are subschemas with custom transformers) in the parsed structure.#2021-11-0506:38Ben SlessThe tldr is cps transform Have a function which returns a parser Every parser gets two continuations, success and failure On providing two continuations you return the actual parsing function Same with decode (you should validate, too while there)#2021-11-0520:50Ben SlessHere, it's a big backwards but it's the most general way of making small building blocks of it
(defn parser
  [parse-fn]
  (fn [success fail]
    (fn [x]
      (try
        (success (parse-fn x))
        (catch Exception e
          (fail e x))))))

(defn coercer
  [schema transformer]
  (let [schema (m/schema schema)
        decoder (m/decoder schema transformer)
        validator (m/validator schema)
        explainer (m/explainer schema)]
    (fn [success fail]
      (fn [x]
        (let [x (decoder x)]
          (if (validator x)
            (success x)
            (fail (explainer x) x)))))))

(def my-parser (parser ,,,))
(def my-coercer (coercer ,,,))

(def work
  (my-parser
   (my-coercer identity on-error)
   on-error))
#2021-11-0507:15robert-stuttaford
(def LearningPathItems
  [:schema {:registry
            {::items
             [:map
              [:items
               {:optional true}
               [:vector
                #_(mu/union LearningPathItemBase)
                [:ref ::items]]]
              ;; marker
              ]}}
   ::items])

#_
(m/validate
 LearningPathItems
 {:items [{:items [{:items []}]}]})
this works. but when i want to introduce another spec to mix into the ref'd spec (to specify lots of other keys that any :items child may have), it fails due to :malli.core/invalid-ref. am i forced to put all of those other key specifications directly in at ;; marker , or is there another way to compose it in? I suppose i could just use Clojure to catenate two vectors but that feels dirty 😅
#2021-11-0507:17robert-stuttafordthis works i guess
(def LearningPathItemBase
  [:map
   [...]])

(def LearningPathItems
  [:schema {:registry
            {::items
             (vec
              (concat
               [:map
                [:items
                 {:optional true}
                 [:vector
                  [:ref ::items]]]]
               (rest LearningPathItemBase)))}}
   ::items])
#2021-11-0507:18robert-stuttafordbtw the new faster mp/provider is fantastic @ikitommi!#2021-11-0710:30ikitommiDiscussion about registries here: https://github.com/metosin/malli/discussions/565#2021-11-0723:51jfntnThe README mentions malli.core/to-ast in https://github.com/metosin/malli#map-syntax but that function does not seem to be around anymore?#2021-11-0805:51ikitommiAdded a note that is is unreleased.#2021-11-1016:17rafdI'm trying to get the properties of a map entry, and struggling. For example, given the following schema, I'd like to access the properties of :widget/id (ie. get back {:foo/bar 123}):
(def Widget
  [:map
   [:widget/id {:foo/bar 123} uuid?]])
I thought this would work, but it returns an invalid-schema error:
(m/properties (m.util/find Widget :widget/id))
Any suggestions?
#2021-11-1818:16ikitommi@U0CLNM0N6 it returns the vector of [key ?props children] as a vector:
(def Widget
  [:map
   [:widget/id {:foo/bar 123} uuid?]])

(mu/find Widget :widget/id)
; => [:widget/id #:foo{:bar 123} uuid?]

(m/children Widget)
; => [[:widget/id #:foo{:bar 123} uuid?]]
#2021-11-1818:16ikitommiyou can call second on it to get the props.#2021-11-1818:48rafdThanks Tommi. I did end up using second, but asked to check if there was a "malli-specific" approach.#2021-11-1818:49ikitommi👍 there are no helpers for handling entries atm. but they are tuple3 always, quite easy to work with 😉#2021-11-1107:07Yehonathan SharvitIs there a way to create a custom Malli tag that will allow me to write:
[:record
   [:user-id :string]
   [:num-of-purchases :int]]
And it would be the same as:
[:and
 [:map-of :keyword :any]
 [:map
   [:user-id :string]
   [:num-of-purchases :int]]
I need this because for the moment JSON decoder doesn't convert :map key strings to keywords https://github.com/metosin/malli/issues/568
#2021-11-1107:15Ben SlessWhy not add your own transformer?#2021-11-1108:46juhoteperi@U0L91U7A8 you could use the json-transformer version I posted on the issue. (You might want to combine with :map-of code from the Malli.transform version)#2021-11-1108:46juhoteperiBut yeah if you create your own registry, you could define :record as :map-of + :map#2021-11-1109:56Yehonathan SharvitHow it would look like?#2021-11-1120:44Nikolas PafitisDoes generate cache the generator?#2021-11-1121:03respatializedmg/generator is what you're looking for I think#2021-11-1121:15Nikolas PafitisYes but if i skip the mg/generator, generate will create a generator, is that generator cached?#2021-11-1818:10ikitomminot yet, but in few mins it does - https://github.com/metosin/malli/pull/575/files#2021-11-1818:14ikitommimg/generator is still bit faster, as there is no cache lookup. Caching is done on Schema instance level, so calling generate with just hiccup effectively bypasses the cache.#2021-11-1818:18ikitommimerged, so generators are also cached by default (with the Schema instance):
(def schema
  (m/schema
    [:map
     [:x boolean?]
     [:y {:optional true} int?]
     [:z [:map
          [:x boolean?]
          [:y {:optional true} int?]]]]))

(comment
  ;; 119µs
  ;; 16µs (cache generator)
  (p/bench (mg/generate schema)))
#2021-11-1818:18ikitommimerged, so generators are also cached by default (with the Schema instance):
(def schema
  (m/schema
    [:map
     [:x boolean?]
     [:y {:optional true} int?]
     [:z [:map
          [:x boolean?]
          [:y {:optional true} int?]]]]))

(comment
  ;; 119µs
  ;; 16µs (cache generator)
  (p/bench (mg/generate schema)))
#2021-11-1322:22ArtemHi folks 👋 is there a way to write a schema for a map that may have any number of string keys, but also a fixed number of keyword keys? (It’s both [:map [:k :type]] and [:map-of :string :type] at the same time)#2021-11-1512:47ikitommicomments welcome on https://github.com/metosin/malli/issues/43#2021-11-1510:34biscuitpantsis it possible to ignore {:optional true} specs when generating values?#2021-11-1511:21biscuitpantsfor anyone wanting to do this, i found this: https://github.com/metosin/malli/blob/master/docs/tips.md#allowing-invalid-values-on-optional-keys which i modified to do what i need!#2021-11-1511:16HukkaWhen would I want to use :sequential instead of :repeat without options? Latter would seem simpler to remember, since I have to use :repeat with min and max now and then.#2021-11-1512:45ikitommiyou can also do [:sequential {:min 1, :max 10}]#2021-11-1512:46ikitommiI would recommend using :sequential always if possible, it’s much faster and always a standalone schema. having :repeat inside of :cat makes it part of the :cat, not a standalone sequence.#2021-11-1513:18HukkaAh, I see, good to know#2021-11-1617:36EddieI am having trouble with mutually recursive schemas raising :malli.core/potentially-recursive-seqex. The intention is for the schemas to be recursive so my question is: why is the exception is being raised? I assume I have expressed recursion in an improper way, but I can’t quite figure it out. https://gist.github.com/erp12/5f7a46ff4a960feeed155da6cf89b0db the schemas and the invocation of m/validate that raises the error. Any help would be much appreciated!#2021-11-1617:44ikitommiIn App, you should wrap the :refs in :schena , like [:schema [:ref :lit]] so that they are not inlined in the :cat.#2021-11-1617:50EddieThanks, that works. I think I understand what you mean by “inlined by the :cat” but I am surprised that :ref must be wrapped. I thought only seqex schemas (`:cat` , :*, :alt, etc) had to be wrapped to prevent inlining. I am probably not understanding something about seqex in general. I’ll keep reading. Thanks again!#2021-11-1815:03rovanionI'm having trouble finding from-ast in malli.core. When I go to source in Emacs I get to 0.7.0-SNAPSHOT, so it should be the right version of the library that's loaded. But I can only see the name declared, never see it def'd.#2021-11-1815:53ikitommireleased most likely next week. You can take the latest sha from master with deps if you want to test it#2021-11-1816:02rovanionSorry, on Leiningen. Will the AST behave like the vector based schemas? That is: Print like a vector/map but not implement ISeq/Associative? What I'm really after is using the Malli schemas I've written to generate some Simple HTML forms.#2021-11-1816:15ikitommiSounds cool, there are multiple ways to walk the schemas for that: 1. use m/walk, look for json schema ns for example how to use it 2. use malli.util/subschemas to get the expanded list of the schema tree. Added that from a malli->form use case 😉 #2021-11-1816:15ikitommion lein, can't use AST yet, sorry#2021-11-1816:15ikitommiAST is just clojure maps#2021-11-1906:38ikitommirefreshed the latest SNAPSHOT with all stuff in:
➜  ~ clj -Sforce -Sdeps '{:deps {metosin/malli {:mvn/version "0.7.0-SNAPSHOT"}}}'
Downloading: metosin/malli/0.7.0-SNAPSHOT/malli-0.7.0-20211118.202503-4.pom from clojars
#2021-11-2209:48rovanionHmm, I get an exception thrown at me when loading malli in my project with this latest snapshot:
#error {
 :cause No such var: mr/fast-registry
 :via
 [{:type clojure.lang.Compiler$CompilerException
   :message Syntax error compiling at (tove/core.clj:1:1).
   :data #:clojure.error{:phase :compile-syntax-check, :line 1, :column 1, :source tove/core.clj}
   :at [clojure.lang.Compiler load Compiler.java 7652]}
  {:type java.lang.ExceptionInInitializerError
   :message nil
   :at [java.lang.Class forName0 Class.java -2]}
  {:type clojure.lang.Compiler$CompilerException
   :message Syntax error compiling at (malli/core.cljc:2362:53).
   :data #:clojure.error{:phase :compile-syntax-check, :line 2362, :column 53, :source malli/core.cljc}
   :at [clojure.lang.Compiler analyze Compiler.java 6812]}
  {:type java.lang.RuntimeException
   :message No such var: mr/fast-registry
   :at [clojure.lang.Util runtimeException Util.java 221]}]
But I don't get the same exception when just requiering malli.core in an empty project with just malli and clojure declared as dependencies.
#2021-11-1815:41Yehonathan SharvitHow can I define a custom predicate? E.g. a string of length 5 (or another more advanced logic).#2021-11-1815:41Yehonathan SharvitI tried the following#2021-11-1815:42Yehonathan Sharvit
[:map {:registry {:asset-id [:and :string #(= 5 (count %))]}}
  [:aa :asset-id]]
#2021-11-1815:42Yehonathan SharvitBut malli says that the schema is not valid#2021-11-1815:52ikitommi@viebel wrap custom fns in :fn#2021-11-1815:54ikitommialso, [:string {:max 6}]#2021-11-1909:11Yehonathan Sharvit:fn is perfect!#2021-11-1909:20Yehonathan SharvitHow would we write a schema for a string made of two components a and b separated by a / where the schema of b depends on the value of a. The valid values of a are known in advance. For instance: 1. When a is "ip" , b should be a valid ip 2. When a is "domain", b should be a valid domain Here are a few examples of valid and invalid data: • ip/127.0.0.1 is valid • ip/111 is not valid • domain/cnn.com is valid • domain/aa is not valid • kika/aaa is not valid#2021-11-1910:36Ben SlessAdd a decoder which splits the string at the separator. Then it's a multi schema for a tuple which dispatches of first#2021-11-1912:33Yehonathan SharvitYou mean a custom transformer like in https://github.com/metosin/malli/blob/master/docs/tips.md#trimming-strings ?
(require '[malli.transform :as mt])
(require '[malli.core :as m])
(require '[clojure.string :as str])

;; a decoding transformer, only mounting to :string schemas with truthy :string/trim property
(defn string-trimmer []
  (mt/transformer
    {:decoders
     {:string
      {:compile (fn [schema _]
                  (let [{:string/keys [trim]} (m/properties schema)]
                    (when trim #(cond-> % (string? %) str/trim))))}}}))

;; trim me please
(m/decode [:string {:string/trim true, :min 1}] " kikka  " string-trimmer)
; => "kikka"
#2021-11-1912:36Yehonathan SharvitBut then, it's the responsibility of the validate callers to explicitly mention my custom transformer. Is there a way to have the custom transformer somehow embedded in the schema?#2021-11-1912:37Ben Sless(def Composite [:tuple {:decode/string #(str/split % #"/") A B])#2021-11-1912:40Ben SlessYou can do some sort of schema constructor, too: (def Composite (-simple-schema (fn [A B] [:tuple {:decode/string #(str/split % #"/") A B])))#2021-11-1912:40Ben SlessThen use it like (Composite [:= domain] Domain)#2021-11-1912:41Ben Slesssort of type-constructors#2021-11-1912:59Yehonathan SharvitThanks @UK0810AQ2 I'll give it a try#2021-11-1913:03Ben SlessThe way I found works well for me is starting from an ideal representation then working backwards with transformers to get there#2021-11-1909:23Yehonathan SharvitIt would be nice to be able to have a distinct error message when we explain why the data is invalid. The explanation could be either: 1. Unknown value of a 2. b is not a valid ip 3. b is not a valid domain#2021-11-1909:32rovanionI don't know how malli works, but if you can use regexes to match strings you could write something like:
((ip)/(:ip-address:)|(domain)/(:domain-name:)) 
#2021-11-1909:33rovanionAnd then look at capture group 1 to see what a is and capture group 2 to find the value.#2021-11-1909:34rovanionIf regexes aren't available the logic that would be transferrable would be to group a and b together in pairs rather than to have separate specs for a and b.#2021-11-2213:25Yehonathan SharvitInside a map, should metadata like title and description be on the field key or in the field value? For instance: [:map [:id {:title "The ID"} :string]] or [:map [:id [:string {:title "The ID"}]] ?#2021-11-2217:35ikitommiboth work, up to you. If the values are shared, pushing the properties into value seem like a better idea, e.g.
(def id (m/schema [:string {:title "The ID"}]))

(m/schema [:map [:id id]])
#2021-11-2303:28Yehonathan SharvitIn most cases the meaning depends on the context. I think it should be given by the key .#2021-11-2217:56ikitommiabout the a/b thing… agree with Ben and that’s how we have used it. the actual domain model could be defined as :multi and there would be encode & decode to&from the string-domain. Actual domain-side would be something like: https://github.com/metosin/malli/blob/master/docs/tips.md#dependent-string-schemas#2021-11-2218:09ikitommiping @viebel.#2021-11-2308:56Yehonathan SharvitThank you @U055NJ5CC for adding a section about string dependent types in malli documentation.#2021-11-2222:05respatializedI'm wondering if there's something I'm not quite understanding about how m/decode and transformers work in the context of :or:
(m/decode
   [:cat {:decode/get {:leave second}} [:= :txt] :string]
   [:txt "something"]
   (mt/transformer {:name :get}))
; => "something"

(m/decode
   [:or [:cat {:decode/get {:leave second}} [:= :txt] :string]
    :double]
   [:txt "something"]
   (mt/transformer {:name :get}))
; => [:txt "something"]
what I'd like is for the :or to decode its value using the decoder provided by the matching schema. I had assumed that decoding was recursive by default, but it doesn't seem to be working in the context of the subschema here.
#2021-11-2305:52Ben SlessI think if you really wanted correct results here you had to write some sort of backtracking search algorithm and end up implementing half of minikanren#2021-11-2315:52respatializedinterestingly, encode does the right thing here:
(m/encode
   [:or [:cat {:encode/get {:leave second}} [:= :txt] :string]
    :double]
   [:txt "something"]
   (mt/transformer {:name :get}))
;; => "something"
what I may have been confused about was just the respective purposes of encoders and decoders
#2021-11-2321:44ikitommiIt works as expected, despite it looks odd. • Decode is a process of turning (potentially invalid) values from external format (e.g. JSON) into values that are valid in the default/clojure domain. • Encode is the opposite: turning valid values from default domain into (potentially invalid) values in another domain (e.g. JSON). here, your decoder is not working correctly, as it emits invalid values. The :or picks the first branch that produces valid values after the transformation. In the example, both paths produce invalid value, so the original value is returned. See the impl: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L713-L724#2021-11-2321:44ikitommiIt works as expected, despite it looks odd. • Decode is a process of turning (potentially invalid) values from external format (e.g. JSON) into values that are valid in the default/clojure domain. • Encode is the opposite: turning valid values from default domain into (potentially invalid) values in another domain (e.g. JSON). here, your decoder is not working correctly, as it emits invalid values. The :or picks the first branch that produces valid values after the transformation. In the example, both paths produce invalid value, so the original value is returned. See the impl: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L713-L724#2021-11-2308:59Yehonathan SharvitMalli decoders are very cool. But as far as I understand, in order to use a schema that relies on decoders, clients have to decode data before validating it. It means that a schema that relies on decoders cannot be used as-is for validating data. Am I correct?#2021-11-2309:19ikitommiFor now, that’s true. There was a discussion some time ago to add :parsed schema element. That would just run m/-parse in before validate, explain, transform etc. like s/conform on spec. Could also be :decoded, so that this would work:
(m/validate [:decoded {:decode :string} schema] "ip/127.0.0.1") ; => true
#2021-11-2309:22ikitommiyou can do that “easily” in the user space too. Just a wrapper-schema basically.#2021-11-2312:36Yehonathan Sharvit@ikitommi Here is my attempt to write a parser in user space:
(defn my-parse
  [schema data]
  (let [data' (decode schema data (transformer string-transformer default-value-transformer))]
    (if (validate schema data')
      (encode schema data' (transformer string-transformer default-value-transformer))
      (humanize (explain schema data')))))
It does the job with asset ids but the problem is that it transform integer to strings.
(def schema 
  [:map
   [:count :int]
   [:asset-id [:multi {:dispatch first
                     :decode/string #(str/split % #"/")
                     :encode/string #(str/join "/" %)}
             ["domain" [:tuple  [:= "domain"] domain]]
             ["ip" [:tuple {:error/message "Invalid IP"} [:= "ip"] ipv4]]]]])

(my-parse schema {:count 12
                  :asset-id "ip/127.0.0.1"})
;; {:count "12", :asset-id "ip/127.0.0.1"}
;; Oops "12" instead of 12
#2021-11-2312:44ikitommijust return the original instead of encode the whole value into string-domain?#2021-11-2312:44ikitommibecause:
(m/encode :int 1 mt/string-transformer) ; => "1"
#2021-11-2312:52ikitommi… or create a custom name for the parsing transformer, e.g. :parse - it’s it a name you have defined, it doesn’t transform anything else.#2021-11-2312:52ikitommi
(let [schema [:map
              [:x :int]
              [:y [:int {:decode/parse (partial + 10)
                         :encode/parse (partial * 2)}]]]
      t (mt/transformer {:name :parse})]
  (as-> (m/decode schema {:x 0, :y 0} t) $
        (m/encode schema $ t)))
;; => {:x 0, :y 20}
#2021-11-2313:13Yehonathan SharvitI cannot return the original as I would like the default values to be added by the parser. For instance, with a schema that assigns 42 as a default value to :count
(def schema 
  [:map
   [:count [:int {:default 42}]]
   [:asset-id [:multi {:dispatch first
                     :decode/string #(str/split % #"/")
                     :encode/string #(str/join "/" %)}
             ["domain" [:tuple  [:= "domain"] domain]]
             ["ip" [:tuple {:error/message "Invalid IP"} [:= "ip"] ipv4]]]]])

(my-parse schema {:asset-id "ip/127.0.0.1"})
;; {:asset-id "ip/127.0.0.1", :count "42"}
#2021-11-2313:20Yehonathan SharvitHowever, I really like the idea of a custom name#2021-11-2313:21Yehonathan SharvitWe need that as our plan is to have a company wide schema repository to be consumed by all apps#2021-11-2321:45Nikolas PafitisIs there an option for MapSchemas to skip generating specific field/s?#2021-11-2321:46ikitomminot atm, but could be. Ideas welcome#2021-11-2321:48ikitommihere’s two:
;; a) a top-level property?
[:map {:gen/fields [:x :y]}
 [:x :int]
 [:y :int]
 [:z :int]]

;; b) entry-level property?
[:map
 [:x :int]
 [:y :int]
 [:z {:gen/gen nil} :int]]
#2021-11-2321:50ikitommib would be 1 line change to the current impl.#2021-11-2322:04Nikolas Pafitisi guess mu/select-keys could be used#2021-11-2322:25Nikolas PafitisI might have encountered a bug:
{:example/user   [:map {:fully/entity?   true
                        :fully.entity/id :user/id}
                  [:user/id :uuid]
                  [:user/username :string]
                  [:user/password :string]
                  [:user/email :string]]}
I have this map and i merge it with default-schemas and schemas to make my registry. Validate and generate work when I use them against
(m/schema type {:registry registry})
but the following returns just nil
(m/properties (m/schema type {:registry registry}))
#2021-11-2322:40Nikolas PafitisIt seems to work if i do deref before properties#2021-11-2322:45Nikolas PafitisIs this expected behaviour?#2021-11-2420:26ikitommiIt's the current behavior and surprises me every time. References are also schemas, and usually without properties. Think like:
(def foo (with-meta {} {:a 1}))
(meta #'foo) ;=> .. no :a here
#2021-11-2420:27ikitomminot 100% happy, but not sure would anything more right than the current behavior#2021-11-2421:31Nikolas PafitisI guess when passed to malli functions like m/properties the different schema types should be dereffed if required. Like in nil punning where nil "can take the correct shape" depending on context#2021-11-2518:32ikitommiwe would assume the references don't have properties, but they could. We could enforce them not to have, but not sure if that would be a good design decision :thinking_face:#2021-11-2420:26ikitommiIt's the current behavior and surprises me every time. References are also schemas, and usually without properties. Think like:
(def foo (with-meta {} {:a 1}))
(meta #'foo) ;=> .. no :a here
#2021-11-2518:32ikitommiwe would assume the references don't have properties, but they could. We could enforce them not to have, but not sure if that would be a good design decision :thinking_face:#2021-11-2416:15respatializedanother encode question: how do I handle subsequences that I want to encode into single values? I've got some data like:
(let [data ["some text" "ex1.csv - file 1" "ex2.csv - file 2"]
      expected-result ["some text" {"file 1" "ex1.csv" "file 2" "ex2.csv"}]]
  (m/encode
    [:catn [:txt [:= "some text"]]
      [:subseq [:repeat {:encode/extract {:leave #(apply merge %)}}
                 [:re {:encode/extract {:enter #(let [[id fnum] (clojure.string/split % #"\s-\s")]
                                                           {fnum id})}}
                      #".* - file \d"]]]]
   data
   (mt/transformer {:name :extract}))
 )

;; => ["some text" {"file 1" "ex1.csv"} {"file 2" "ex2.csv"}]
the #(apply merge %) encoder works to merge the maps extracted if the :repeat schema is a top-level schema, but not if it is a schema for a subsequence of a seqex. am I pushing encode too far here? should I be relying on parse to do this kind of grouping?
#2021-11-2421:17respatializedthinking about this a bit more, it doesn't seem like this is possible, as different values are being encoded depending on whether the sequex is top-level or a subsequence: top-level:
(let [data ["ex1.csv - file 1" "ex2.csv - file 2"]
      expected-result {"file 1" "ex1.csv" "file 2" "ex2.csv"}]
  (m/encode
     [:repeat {:encode/extract {:leave #(apply merge %)}}
                 [:re {:encode/extract {:enter #(let [[id fnum] (clojure.string/split % #"\s-\s")]
                                                           {fnum id})}}
                      #".* - file \d"]]
   data
   (mt/transformer {:name :extract}))
 )
;; => {"file 1" "ex1.csv", "file 2" "ex2.csv"}
so really the object of encoding is a complete vector, the vector of maps created after each string element gets encoded. whereas with the subsequence, it's not a clearly delineated individual value:
["some text"
#_"start of subsequence in :repeat"  
 {"file 1" "ex1.csv"} {"file 2" "ex2.csv"}
... #_"arbitrary number of additional repeats"
 #_"end of subsequence in :repeat" ]
and thus cannot be passed in as an argument to the function referenced in :encode/extract , even if that function accepts varargs or an input sequence. so I think I need to rely on m/parse to do the grouping I am hoping for here.
#2021-11-2418:26Adam HelinsIs there some way for having context sensitive transformers? e.g. when transforming a child in a coll, being able to take into account what happened to already transformed children#2021-11-2509:11Adam HelinsIn other words, is transforming strictly context-free?#2021-11-3011:52ikitommicould you provide an minimalistic example when this would be useful. There was a discussion about this long ago, but a) there was no real-life examples for “why?” and b) context-free made performance optimization much easier. Happy to revisit, but need the a) 🙂#2021-11-3011:54ikitommithere was also a discussion just to implement all applications (validation, transforming) using walk. That was my original idea for Spec too (https://clojure.atlassian.net/browse/CLJ-2251).#2021-11-3011:55ikitommi4 years! time flies 🙂#2021-11-3011:55ikitommi> Created 11 October 2017, 22:22#2021-11-3011:52ikitommicould you provide an minimalistic example when this would be useful. There was a discussion about this long ago, but a) there was no real-life examples for “why?” and b) context-free made performance optimization much easier. Happy to revisit, but need the a) 🙂#2021-11-2504:48JohanWould malli be a good place to check if value is within a list of ~20k items ? Build a giant [:enum] or custom fn or just not use malli for that ?#2021-11-2509:44dangercoderI would put that list of 20k items into a set instead and use clojure.core contains? function. If you want to check that through malli you can always create a custom :fn#2021-11-2512:52JohanFor example I want to check a postcode, would you just use a malli schema to check if the value is a number and then check if it's part of the country list of postcode ?#2021-11-2515:03vinurs
(def get-categories-req
  [:map {:title "get-categories-req"}
   [:categories  [:vector {:description "category id"
                           :default [0 1]
                           } int? ]]])
hello, i define the malli schema like this, and call it shows the error
{
  "schema": "[:map {:title \"get-categories-req\", :closed true} [:categories [:vector {:description \"category id\", :default [0 1]} int?]]]",
  "errors": [
    {
      "value": "0,1",
      "in": [
        "categories"
      ],
      "message": "invalid type",
      "path": [
        "categories"
      ],
      "schema": "[:vector {:description \"category id\", :default [0 1]} int?]",
      "type": "malli.core/invalid-type"
    }
  ],
  "value": {
    "categories": "0,1"
  },
  "type": "reitit.coercion/request-coercion",
  "coercion": "malli",
  "in": [
    "request",
    "query-params"
  ],
  "humanized": {
    "categories": [
      "invalid type"
    ]
  }
}
how can i make malli parse this?
#2021-11-2516:57Ivan FedorovCan malli’s [:merge] work inside a registry? this fails for me
(def registry:shopify
  {:e.shopify/address1
   [:map [:first_name string?]]
   :e.shopify/address2
   [:map [:city string?]]
   :e.shopify/address-full
   [:merge
    :e.shopify/address1
    :e.shopify/address2]})

(def schema:address-full
  (m/schema
    [:schema {:registry registry:shopify}
     :e.shopify/address-full]))
#2021-11-2518:35ikitommiDo you have the :merge registered? It's not in the default registry (just yet at least)#2021-11-2611:47Ivan Fedorov@U055NJ5CC How would I do this? This doesn’t work at least 🙂
(def registry:shopify--
  {:merge mu/merge
   :e.shopify/address1
          [:map [:first_name string?]]
   :e.shopify/address2
          [:map [:city string?]]
   :e.shopify/address-full
          [:merge
           :e.shopify/address1
           :e.shopify/address2]})
#2021-11-2708:45ikitommimaps do not have order, any of the entries using merge might get resolved before the :merge. I’ll change how the default registry works, so this would be easier. Before that: 1. use mu/merge instead of :merge 2. swap the default registry with one that has the :merge 3. use composite registry, e.g. (mr/composite-registry {:merge mu/merge} {:e.shopify/address1 …}), in case the :merge should be registered before the actual schemas#2021-11-2616:01emccueWhats the best way to add documentation to a schema#2021-11-2616:01emccue
[:map {:doc "A position"}
  [:x {:doc "The x coordinate"} :int]
  [:y {:doc "The y coordinate"} :int]]
#2021-11-2616:02emccuei’m fine making something up, but if there is a “standard” schema property i would want to use it#2021-11-2619:55ikitommi:title and :description are used for JSON SCHEMA. Kinda long, but current.. standard#2021-11-2619:55ikitommihttps://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc#L16#2021-11-2713:13ikitommiallowing schema registry to be swapped without compiler/jvm-options: https://github.com/metosin/malli/pull/583. There would be a "strict" mode too, if this is too loose for someone. But might be better for the 90% of cases?
(require '[malli.core :as m]
         '[malli.util :as mu]
         '[malli.registry :as mr]
         '[malli.generator :as mg])

;; look ma, just works
(mr/set-default-registry!
  (mr/composite-registry
    (m/default-schemas)
    (mu/schemas)))

(mg/generate
  [:merge
   [:map [:x :int]]
   [:map [:y :int]]])
; => {:x 0, :y 92}
comments welcome.
#2021-11-2713:14ikitommiping @U0A5V8ZR6#2021-11-2713:16ikitommiyour sample could be:
(mr/set-default-registry!
  (mr/composite-registry
    (m/default-schemas)
    (mu/schemas)
    {:e.shopify/address1 [:map [:first_name string?]]
     :e.shopify/address2 [:map [:city string?]]
     :e.shopify/address-full [:merge
                              :e.shopify/address1
                              :e.shopify/address2]}))

(mg/generate :e.shopify/address-full)
;{:first_name "0GiD"
; :city "DMS5wc6mXcN4JCp9lJ"}
#2021-11-2713:17ikitommialso, you could inject a mutable-registry as last in the registry-chain and add the schemas using custom register! fn. Mutate like a boss 🙂#2021-11-2714:07Ivan FedorovOh this looks sweet! Thank you, mr Reiman!#2021-11-2717:08pithylessLooking forward to this; I’ve ended up fighting the tooling too often trying to get the compiler options working correctly in a consistent way.#2021-11-2718:11ikitommimerged in master. a blog post and shipping 0.7.0 out.#2021-11-2714:26Karol WójcikWill malli support records?#2021-11-2714:45ikitommiwhat would that look like?#2021-11-2722:53emccue
[:and [:fn #(instance? RecordType %)]
      [:map [:x :int] [:y :int]]]
Is a basic way if you just need validation and not generation
#2021-11-2722:55emccuei’m not clever enough to compose the :map generator with map->RecordType but i assume there is a way#2021-11-2808:46Karol WójcikI was thinking about something like schema is doing.#2021-11-2809:57ikitommim/defrecord I presume. I think Malli should have that and m/defproticol too.#2021-11-2814:25Karol WójcikCool. Looking forward to this.#2021-12-0115:53wcalderipeCan I get the schema of a :multi by passing the dispatched value to a function? Example:
(def Multi
    [:multi {:dispatch :type}
     [:foo [:map [:value :int]]]
     [:bar [:map [:value :string]]]])

  (get-schema Multi {:type :foo}) ;; => [:foo [:map [:value :int]]]
#2021-12-0117:02ikitommi@wcalderipe, maybe:
(def Multi
  [:multi {:dispatch :type}
   [:foo [:map [:value :int]]]
   [:bar [:map [:value :string]]]])

(defn get-schema [schema value]
  (some-> schema
          m/properties
          :dispatch
          (apply (list value))
          (->> (mu/get schema))))

(get-schema Multi {:type :foo})
; => [:map [:value :int]]
#2021-12-0117:04ikitommiif you want the whole entry, use mu/find:
(defn get-schema [schema value]
  (some-> schema
          m/properties
          :dispatch
          (apply (list value))
          (->> (mu/find schema))))

(get-schema Multi {:type :foo})
; => [:foo nil [:map [:value :int]]]
#2021-12-0117:05ikitommiit’s always tuple3 of [key properties value]#2021-12-0207:38wcalderipe@U055NJ5CC thanks a bunch for the help 🙌#2021-12-0316:49Ben SlessIdea - charset predicate for string schemas (alpha, alphanumeric, etc)#2021-12-0509:18Ben Slesshttps://github.com/metosin/malli/pull/587#2021-12-0520:25ikitommiLook good. The cljs-support would be really good as I think these will be used a lot.#2021-12-0520:26ikitommiany change of adding those too?#2021-12-0520:40Ben SlessTheoretically, yes, practically, my knowledge in cljs is limited and we've seen how some of the techniques I'm used to from clj don't work well for cljs. Also worried about code size This will require more careful review on your end, but I don't mind giving it a shot#2021-12-0521:34Ben SlessOkay, my cljs tests are failing and setting up a REPL environment is a pain. Will leave it to tomorrow#2021-12-0609:11Ben Sless@U055NJ5CC added cljs implementation#2021-12-0610:03ikitommipartywombat commented.#2021-12-0610:15Ben SlessAgreed with most comments but I think there are some edge cases regarding blankness and nil#2021-12-0509:18Ben Slesshttps://github.com/metosin/malli/pull/587#2021-12-0716:15ikitommi,🥳#2021-12-0716:55dharriganAmazing! Such a great library to use!#2021-12-0717:00Ben SlessIt's also a pleasure to hack on#2021-12-0717:00Ben SlessIt's also a pleasure to hack on#2021-12-0721:16dev-hartmannHey folks I'm new to malli and really blown away by what it can do. I just have a small question I couldn't figure out from the readme ( maybe I'm just blind) can I parse a json Schema to a malli schema?#2021-12-0721:31Ben SlessNot yet 🙂#2021-12-0721:32Ben SlessI started working in it but it's missing a few bits and bobs on the JSON schema specification to be happy with#2021-12-0814:03Yevgeni TsodikovHey, I see that https://clojars.org/metosin/malli/versions/0.7.0T exists in Clojars. What’s the difference between that and https://clojars.org/metosin/malli/versions/0.7.0?#2021-12-0814:21ikitommiIt's a typo release.#2021-12-0815:06Yevgeni TsodikovIt’s messing up with https://github.com/renovatebot/renovate, it thinks it’s a later release than 0.7.0 😞#2021-12-0816:17ikitommigood thing is that the contents are identical to 0.7.0.#2021-12-0816:17ikitommigood reason to ship the next version soon.#2021-12-0816:26Ben SlessEven just a point release to park on the latest artifact id#2021-12-0907:18Ben SlessThis is very cool https://github.com/erp12/schema-inference#2021-12-0911:51Ben Sless@U7XR2PZFW#2021-12-0914:43EddieThanks! I’m surprised you found this so quickly without me doing any advertising. :) I am currently using this prototype in a research project to give a more concrete example of how something like this could be valuable. After I make that work public, I plan to make a wider pitch to the community. That said, if people have any feedback and ideas now, I would love to hear it!#2021-12-0914:44Ben SlessFirst of all, you get cool-points 😎#2021-12-0914:45Ben SlessSecond, can you share some of your design decisions? Why did you choose a bespoke representation rather than use tools.analyzer output?#2021-12-0914:46Ben SlessAnother thing I thought of while reading the code was some impedance mismatch between the type information and the expression encoding which could be unified#2021-12-0914:49Ben SlessYou could create "big lambda"
;; Type Abstraction
'[:FN [:cat T] U]
Then the abstraction and environment representations could be shared
#2021-12-0914:50Ben SlessAlthough => is pretty much that, already In its context, why do you need to tag s-vars at all? You know it's a function of type to type#2021-12-0915:02Ben SlessYou can blame someone I follow for starring your project 🙂#2021-12-0916:16EddieThis is all great. I appreciate you taking the time to take a closer look. > Why did you choose a bespoke representation rather than use tools.analyzer output? This is the first I’m hearing of tools.analyzer :) It seems ideal for this kind of thing. I stuck to a representation that was as close to lambda calculus and Hindley–Milner as possible to make the papers easier to translate into code. The thought of expanding the bespoke representation to cover all of Clojure gave me nightmares. Tools.analyzer might save the day. > Another thing I thought of while reading the code was some impedance mismatch between the type information and the expression encoding which could be unified … > Although => is pretty much that, already Yea, figured it would be nice if everyone’s function schemas “just worked” as function types during inference without any changes, so I used :=>. I’m sure you noticed that only positional args are supported right now (via :cat). At some point I would like to try to add support for varargs. Also maybe destructuring, but that might be challenging. Still, you raise a good point about the unnecessary separation between expressions trees and type/schema trees. As you mentioned, the tagging of :var and :s-var could be eliminated. Also the logically equivalent functions like substitute-vars and substitute-types could be made into 1 generic function. I went back and forth on this design decision a couple times. In the end, my decision to separate into 2 kinds of trees was motivated by wanting more explicit information about what went wrong when debugging these algorithms. I was brand new to type theory when I started. With hindsight, I wouldn’t choose the same design. Based off of 1 mornings worth of googling, I think a change to tools.analyzer and a unified encoding is a good direction!#2021-12-0916:24Ben SlessThe only problem with tools analyzer is it has a lot of output which can get unwieldy, there are about 15 node types to parse there instead of 4#2021-12-0916:25Ben SlessThen you might end up needing to use another method to walk the tree, either the analyzer's walk mechanism with multi methods or something like meander#2021-12-0916:26Ben SlessIf you lift the type representation to ast nodes as well you could then write your own emitter which translates it back down to a schema#2021-12-0907:33Nechemya UngarHi all, I am new to clojure (using clojurescript) and I come form a static-typing background (typescript), I looked into bringing some order into the code base I got into, and malli was recommended to me over spec. I wanted to say that for newcomers like me it would have been helpful if the readme actually started with introduction of the malli provider - it really helps to create a quick scaffolding for data structures. thanks2#2021-12-0912:55amithgeorgeIn the readme, it mentions decimal? as built-in schema. Checking the relevant https://github.com/metosin/malli/blob/0.7.0/src/malli/core.cljc#L2234, decimal? is not registered as a schema. decimal? was removed in this https://github.com/metosin/malli/commit/5deae59357b2eb3fb9890282e679bb2d7fa557e3. Is it possible to add it back? At the moment I am using decimal? in a :fn schema and that works.#2021-12-0915:31ikitommiPR welcome to add it back#2021-12-0916:24amithgeorgeI maybe missing something here. I added decimal? to predicate-schemas. My expectation was after that evaling the file with this change, the following code should work
(validate :decimal 10.9M)
But that errors out with this message
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
; :malli.core/invalid-schema {:schema :decimal}
For context,
(validate decimal? 10.9M)
works fine with the change. What else needs to change to support :decimal . Also, do you think this is needed?
#2021-12-0916:27amithgeorgeFound it. If :decimal support is to be added, then I need to add :decimal to type-schemas . I will exclude this change for now and raise a PR for only the inclusion of decimal? .#2021-12-0916:51amithgeorgePR - https://github.com/metosin/malli/pull/591#2021-12-0917:14ikitommiMerged, thanks!#2021-12-1012:07Ben SlessThinking a bit about a workable composition of transformer and validator, will it be be correct to model it via a cps transform where validation is called after leave stage for each element? It is true that a schema could transform its children arbitrarily, so how could this be done in a single pass? Feels like trying to solve the halting problem#2021-12-1013:14ikitommiwoudn't the validation be called a lot more that actually needed? e.g. leaf validates itself, one level up needs to re-validate the leaf as part of it's own validation, etc.#2021-12-1013:18ikitommithe prismatic approach: https://plumatic.github.io//schema-0-2-0-back-with-clojurescript-data-coercion#2021-12-1013:31Ben SlessIdeally, no, you assume the leaves are already correct if you made it to the parent then you just satisfy the parent condition or combinator#2021-12-1013:35Ben SlessLet's say that each coercer has a success or failure continuation. On failure, we just return, on success, we call the transformation and validation of the next value#2021-12-1013:36Ben SlessThen you need to define the combination semantics for map, sequence, vector, tuple, and, or#2021-12-1013:37Ben SlessMaybe the validation itself can be split to enter/leave#2021-12-1013:37Ben SlessMap could check on enter that it's a map, then coerce its children#2021-12-1013:40Ben SlessThis holds as long as a transformer doesnt modify children recursively#2021-12-1013:40Ben SlessWhich is true for 99% of cases? So maybe there could be an optimistic coercer which isnt guaranteed to work all the time#2021-12-1100:00steveb8nQ: I have a single arg fn that I am trying to generatively test. The single arg is vector that is checked using a :catn schema. when I use the => syntax, it interprets the nested :catn schema for the fn arity instead of treating it as a single arg. has anyone figured out how to do this?#2021-12-1100:43emccue[:cat [:cat :int :int]]#2021-12-1100:43emccueall fn args are cat#2021-12-1100:43emccueoh wait no i just realized your issue#2021-12-1103:27steveb8nYeah. Simple but not easy. I can work around it by wrapping with another fn with a map arg.#2021-12-1103:27steveb8nI'll log an issue to see if we are missing something#2021-12-1103:53ikitommiwrap in :schema to push out of the sequence#2021-12-1105:01steveb8nthx @U055NJ5CC I’ll give that a try#2021-12-1121:14steveb8nconfirming, using :schema around the vector schema fixes the problem#2021-12-1107:20Ben SlessReading @afoltzm FSM post again, I returned to the idea of some sort of schema router / compiler, where an or of schemas will pull out the common denominator then dispatch to the difference Thoughts?#2021-12-1109:39steveb8ncan you explain more? I’ve been using them more in SPA’s so interested in this#2021-12-1110:25Ben SlessImagine [:or [:map [:a any?] [:b any?]] [:map [:a any?] [:c any?]]] => [:and [:map [:a any?]] [:or [:map [:b any?]] [:map [:c any?]]]] to start with, even smarter to have a dispatch to the correct schema instead of or#2021-12-1115:23respatializedSounds like a malli.diff namespace might be an interesting idea, because this concept may be applicable beyond just :or schemas – for example, you'd be able to compare two seqex schemas and say things like "sequence B can contain all the same elements as sequence A except for elements X and Y"#2021-12-1115:26Ben SlessI'm just getting started, [:or :int :number] => :number#2021-12-1115:27Ben Slessschemas as segments and sets#2021-12-1120:02mynomotoCan I have a link for this post? It looks interesting.#2021-12-1120:04Ben SlessSure, https://fabricate-site.github.io/fabricate/finite-schema-machines.htmlhtml @mynomoto#2021-12-1120:45mynomotoThis is really interesting, both the post and the routing idea 👏#2021-12-1121:22ikitommijust would have needed an intersection of two map schemas yesterday.#2021-12-1121:24ikitommipluggable optimizer is a great idea, here's one stab at it: https://github.com/miikka/boolean-simplifier#2021-12-1209:32Ben Sless@U7XR2PZFW's work on inference could intersect with this as well. Every type of/or schema can represent a set of predicates. A predicate can be a singular predicate, a segment (upper and lower bounds) or finite domain (min max)#2021-12-1219:49EddieSounds like roughly equivalent to sub-typing for structural types. If we assume collections are covariant with respect to elements, functions are contravariant with respect to arguments, and a few other assumptions, I suspect small-ish rule system could be used to implement a sub-type predicate.#2021-12-1219:52Ben SlessWe can even unpack map schemas to a closed set of predicates, being map? and a set of all the entry predicates. Same for tuples. Then conjunction and disjunction are pretty well defined and we remain in predicate land and not structure. Not sure if it's good#2021-12-1220:10Ben SlessRoughly:
(defprotocol SchemaDomain
  (-conj [this that]) ;; and
  (-disj [this that]) ;; or
  (-isa? [this that])
  (-walk [this]))

(defrecord EntrySchema [k v]
  (-conj [this that]
    (EntrySchema. k (-conj v (:v that))))
  (-disj [this that]
    (EntrySchema. k (-disj v (:v that)))))

(defrecord PredicateSchema [p]
  (-conj [this that]
    (cond
      (-isa? this that) this
      (-isa? that this) that
      :else (-and this that)))
  (-disj [this that]
    (cond
      (-isa? this that) that
      (-isa? that this) this
      :else (-or this that))))
#2021-12-1119:43respatializedI've encountered an issue where sequence schemas appear not to preserve metadata of input collections:
(meta (m/decode
         [:schema {:decode/get {:enter identity}} [:cat [:= :start] :map]]
         (with-meta [:start {:a 2}] {:meta true})
         (mt/transformer {:name :get})))
;; => nil

(meta (m/encode
         [:schema {:encode/get {:enter identity}} [:cat [:= :start] :map]]
         (with-meta [:start {:a 2}] {:meta true})
         (mt/transformer {:name :get})))
;; => nil
is this expected behavior, or should malli preserve this information?
#2021-12-1119:44respatialized(happy to file an issue on GitHub if it would be helpful)#2021-12-1120:04Ben SlessSure, https://fabricate-site.github.io/fabricate/finite-schema-machines.htmlhtml @mynomoto#2021-12-1208:27ikitommirelated to is-a relations, malli should have more concrete type definitions in the core, e.g. pos-int? is not a type, it’s a :int with constraint of being positive. Less types to map in transformers and just better. Related: • https://github.com/metosin/malli/issues/264https://github.com/clj-kondo/clj-kondo/blob/d9fca2705863e3e604e004ccb942e0b3d2e268ec/src/clj_kondo/impl/types.clj#L18-L51#2021-12-1216:10ikitommion master - adding type-hints for providers, starting with :map-of:
(require '[malli.provider :as mp])

(mp/provide
  [^{::mp/hint :map-of}
   {:a {:b 1, :c 2}
    :b {:b 2, :c 1}
    :c {:b 3}
    :d nil}])
;[:map-of
; keyword?
; [:maybe [:map
;          [:b int?]
;          [:c {:optional true} int?]]]]
#2021-12-1216:11ikitommithe current threshold with unified key & values schemas is 3, so:
(mp/provide
  [{:a [1]
    :b [1 2]
    :c [1 2 3]}])
; [:map-of keyword? [:vector int?]]
#2021-12-1216:12ikitommi… can be configured via options.#2021-12-1216:12ikitommiproviders start to be useful 🙂#2021-12-1220:50ikitommiadding :tuple inferreing, provider already 89 loc 🙀#2021-12-1220:52ikitommiwith :malli.provider/tuple-threshold defaulting to 3:
;; tuple-like with too few elements
   [[:vector some?]
    [[1 "2" true]
     [2 "2" true]]]

   ;; tuple-like with enough samples
   [[:tuple int? string? boolean?]
    [[1 "2" true]
     [2 "2" true]]
    {::mp/tuple-threshold 2}]

   ;; tuple-like with enough samples
   [[:tuple int? string? boolean?]
    [[1 "2" true]
     [2 "2" true]
     [3 "3" true]]]

   ;; tuple-like with non-coherent data
   [[:vector some?]
    [[1 "2" true]
     [2 "2" true]
     [3 "3" "true"]]]

   ;; a homogenous hinted tuple
   [[:tuple int? string? boolean?]
    [^{::mp/hint :tuple} [1 "2" true]
     [2 "2" true]]]

   ;; a hererogenous hinted tuple
   [[:tuple int? string? some?]
    [^{::mp/hint :tuple} [1 "2" true]
     [2 "2" "true"]]]

   ;; invalid hinted tuple
   [[:vector some?]
    [^{::mp/hint :tuple} [1 "2" true]
     [2 "2" true "invalid tuple"]]]
#2021-12-1220:53ikitommihttps://github.com/metosin/malli/pull/593#2021-12-1621:45borkdude@ikitommi The newest clj-kondo (just released) supports automatically loading configs from .clj-kondo/*/*/config.edn - just saying :)#2021-12-1810:37ikitommithis is great news 🥳 !!#2021-12-1812:48ikitommi@U04V15CAJ so, this looks right? https://github.com/metosin/malli/pull/598#2021-12-1812:49ikitommifile-writing looks like:
#?(:clj
   (defn save! [config]
     (let [cfg-file (io/file ".clj-kondo" "configs" "malli" "config.edn")]
       (io/make-parents cfg-file)
       (spit cfg-file config)
       config)))
#2021-12-1812:50borkdudeit's recommended to write a directory that looks like:
<org>/<lib>/config.edn
#2021-12-1812:50borkdudeso then it would be:
<metosin>/<malli>/config.edn
#2021-12-1812:50borkdudebut this is usually for configs about malli and not provided by malli about other things#2021-12-1812:51borkdudeso maybe this would be better and would still give room for malli itself to export config for its own macros too:
<metosin>/<malli-type-anns>/config.edn
#2021-12-1812:52borkdudebut you are correct that the config isn't necessary anymore on the user's behalf#2021-12-1820:23ikitommifor some reason, the auto-loading doesn’t work for me :thinking_face:#2021-12-1820:23ikitommirepro: https://github.com/metosin/malli/pull/598#2021-12-1820:23borkdudehave you upgraded clj-kondo#2021-12-1820:24ikitommi
➜  malli git:(clj-kondo-config-fix) ./bin/kaocha --focus malli.clj-kondo-test
[(.....)]
1 tests, 5 assertions, 0 failures.

➜  malli git:(clj-kondo-config-fix) cat .clj-kondo/configs/metosin/malli/config.edn
{:linters {:unresolved-symbol {:exclude [(malli.core/=>)]}, :type-mismatch {:namespaces {malli.clj-kondo-test {kikka {:arities {1 {:args [:int], :ret :int}, :varargs {:args [:int :int {:op :rest, :spec :int}], :ret :int, :min-arity 2}}}, siren {:arities {2 {:args [:ifn :coll], :ret :map}}}}}}}}%

➜  malli git:(clj-kondo-config-fix) clojure -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "2021.12.16"}}}' -M -m clj-kondo.main --lint test/malli/clj_kondo_test.cljc
linting took 163ms, errors: 0, warnings: 0

➜  malli git:(clj-kondo-config-fix) echo '{:config-paths ["configs/metosin/malli"]}' > .clj-kondo/config.edn

➜  malli git:(clj-kondo-config-fix) clojure -Sdeps '{:deps {clj-kondo/clj-kondo {:mvn/version "2021.12.16"}}}' -M -m clj-kondo.main --lint test/malli/clj_kondo_test.cljc
test/malli/clj_kondo_test.cljc:80:9: error: Expected: integer, received: string.
linting took 200ms, errors: 1, warnings: 0
#2021-12-1820:24ikitommiI think that’s the latest one#2021-12-1820:25borkdudeyou should use the pattern .clj-kondo/*/*/config.edn , you have one dir too many#2021-12-1820:25ikitommioh, it was not **! 🙂#2021-12-1820:25borkdude.clj-kondo/metosin/malli#2021-12-1820:27borkdudebut I would rename this dir, as I said#2021-12-1820:27ikitommiworks like a charm, 🙇#2021-12-1820:27ikitommiyes, I will, just trying it out#2021-12-1820:27borkdude.clj-kondo/metosin/malli-types or so#2021-12-1820:27borkdudeso you can still export config for malli itself at one point#2021-12-1820:30ikitommiyes, changed that, will merge when tests pass. This is a really good change, makes the newbie user experience so much better.#2021-12-1820:30borkdudeI hope so.. :)#2021-12-1810:38ikitommimore batteries for value type inferring (https://github.com/metosin/malli/pull/597):
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])

(mp/provide
 [{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"}
  {:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"}])
; => [:map [:id string?]]

(mp/provide
 [{:id "caa71a26-5fe1-11ec-bf63-0242ac130002"}
  {:id "8aadbf5e-5fe3-11ec-bf63-0242ac130002"}]
 {::mp/value-providers {'string? {:uuid mt/-string->uuid}}})
; => [:map [:id :uuid]]

(mp/provide
 [{"0423191a-5fee-11ec-bf63-0242ac130002" {:id "0423191a-5fee-11ec-bf63-0242ac130002"}
   "09e59de6-5fee-11ec-bf63-0242ac130002" {:id "09e59de6-5fee-11ec-bf63-0242ac130002"}
   "15511020-5fee-11ec-bf63-0242ac130002" {:id "15511020-5fee-11ec-bf63-0242ac130002"}}]
 {::mp/value-providers {'string? {:uuid mt/-string->uuid}}})
; => [:map-of :uuid [:map [:id :uuid]]]
#2021-12-1821:59respatializedI am encountering an arity error when I try to create a function schema for a multi-arity fn that accepts a vector of a specific type as an input.
(m/schema
 [:function
  [:=> [:cat [:schema [:* :int]]] :any]
  [:=> [:cat [:schema [:* :int]] :boolean] :any]])
This produces the following error (I confirmed this with the latest tagged release of malli, 0.7.4):
{:type :malli.core/duplicate-arities,
    :message :malli.core/duplicate-arities,
    :data
    {:infos
     [{:min 0, :arity :varargs, :input [:cat [:schema [:* :int]]], :output :any}
      {:min 1,
       :arity :varargs,
       :input [:cat [:schema [:* :int]] :boolean],
       :output :any}]}}
I had figured that wrapping the input sequence schema in :schema would mark it as a distinct sequential value rather than a subcomponent of the input arg vector, but it appears that malli is treating an input vector of an arbitrary size as indicating that the function is supposed to have varargs, resulting in a clash between the arities of the function.
#2021-12-1822:07respatializedrunning mg/generate on the arity input seqexes independently produces values that I expect to see:
(mg/generate [:cat [:schema [:cat [:* :int]]] :boolean])
;; => ((-2 2096836 3803 45 -49393) false)
(mg/generate [:cat [:schema [:cat [:* :int]]]])
;; => ((-1709 -2289 -38 3962880 -8 -948 13075997 77734 -16118422 635358 2 -1 -1))
#2021-12-1908:38ikitommiLooks like :schema has a bug related to m/-regex-min-max :
(m/-regex-min-max
 (m/schema
  [:cat [:schema [:* :int]]]))
; => {:min 0}
could you write an issue out of this?
#2021-12-1908:39ikitommiwhile waiting, you can try:
(m/-regex-min-max
 (m/schema
  [:cat [:sequential :int]]))
; => {:min 1, :max 1}
#2021-12-1908:41ikitommi.. and if it’s a vector, this is a way to describe it:
(m/schema
 [:function
  [:=> [:cat [:vector :int]] :any]
  [:=> [:cat [:vector :int] :boolean] :any]])
ping @UFTRLDZEW
#2021-12-1908:41ikitommibut, please issue the original, as it’s a bug.#2021-12-2015:11respatializedhttps://github.com/metosin/malli/issues/601 @U055NJ5CC thanks for taking a look.#2021-12-1822:07respatializedrunning mg/generate on the arity input seqexes independently produces values that I expect to see:
(mg/generate [:cat [:schema [:cat [:* :int]]] :boolean])
;; => ((-2 2096836 3803 45 -49393) false)
(mg/generate [:cat [:schema [:cat [:* :int]]]])
;; => ((-1709 -2289 -38 3962880 -8 -948 13075997 77734 -16118422 635358 2 -1 -1))
#2021-12-1906:38Nechemya Ungarjava.lang.RuntimeException: Unable to resolve symbol: qualified-keyword? in this context, compiling:(malli/core.cljc:167:51) Hi all: I have run into this error last week, at the beginning lein clean would resolve the issue but now our clojurescript project would not compile with malli . If we add malli to our dependencies in project.clj and require malli.core the compiler spits out the aforementioned error. Is there a minimum clojure/clojurescript version? this is our versions:
[org.clojure/clojurescript "1.10.439"]
[org.clojure/clojure "1.8.0"]
Thank you 🙏
#2021-12-1908:44juhoteperiQualified-keyword? was added in 1.9#2021-12-1908:47Nechemya Ungarwow, thank you for pointing it out (I am new to clojure) so I wonder why it worked in the 1st place#2021-12-1909:06juhoteperiWell, you have the Cljs 1.10 so that does have qualified-keyword. If you aren't using Malli in Clojure side, the problem could be macro use from Cljs and perhaps you changed something that caused Malli to be required in Cljs macro compilation time (so in Clj side)?#2021-12-1909:07juhoteperiAnd btw. Malli requires 1.10, there is another case why 1.9 isn't enough.#2021-12-1909:33juhoteperiReadme mentions 1.10 now#2021-12-1910:46Ben Slesstangential question, why were you using 1.8 to begin with?#2021-12-1910:53Nechemya UngarWell I came into the company half a year ago, I was doing typescript before... So I didn't touch anything related to the config#2021-12-1910:53Nechemya UngarThe code base is immense I thought using some schemas can help and malli was recommended to me over spec#2021-12-1909:23ikitommireleased [metosin/malli "0.7.5"], a fifth patch since 0.7.0, accumulated changes: • https://github.com/clj-kondo/clj-kondo/blob/master/CHANGELOG.md#20211216 can load malli type configs automatically from new location (`.clj-kondo/metosin/malli-types/config.edn`), thanks @borkdude. • use https://github.com/brandonbloom/fipp for fast pretty-printing the clj-kondo configs • updated dependencies • schema inferring supports value decoding via options • :map-of inferring can be forced with `:malli.provider/hint :map-of` meta-data • :tuple inferring (supports type-hints and threshold options) • FIX Function with Sequential return value cannot define as function schema • FIX `decimal?` predicate schema was removed in 0.7.0#2021-12-2202:34steveb8nQ: I have a typescript project which can expose react prop types. I’m wondering if there’s a way to convert these types to Malli using some kind of tooling? I don’t mind building the tool and submitting a PR if there is a way. Any typescript experts have ideas for this?#2021-12-2203:23steveb8nbackground: I have a contractor building react components in Storybook/typescript which I use in a re-frame app. This workflow is huge win#2021-12-2203:23steveb8nif I can figure out types -> Malli, I can enhance the testing story as well#2021-12-2205:19steveb8nI’m a typescript/babel noob so I need to figure out how to expose the types and then interop to generate Malli#2022-12-2716:59ikitommithere is https://github.com/tiagodalloca/malli-ts, is that still worked on @U4U6BDQTE ?#2022-03-0614:07Tiago Dall'Ocahey!#2022-03-0614:08Tiago Dall'OcaI need to show up here more often haha#2022-03-0614:08Tiago Dall'Ocayess, I'm working on it 🙂 (slowly)#2021-12-2214:20Ivan FedorovIs this normal that :e.order/shipping doesn’t resolve into a schema here? Or maybe I lost some understanding
(def registry:case
  {:e.order/shipping
   [:map [:address string?]]
   :e/order
   [:map
    [:shipping-lines
     [:vector [:ref :e.order/shipping]]]]})

(def schema:customer-order-case
  (m/schema
    [:schema
     {:registry registry:case}
     :e/order]))

(comment
  (-> (m/deref-all schema:customer-order-case)
      (m/children)
      (last) (last)
      (m/children) first ; -> [:ref :e.order/shipping]
      (m/children) first type))

;; comment block evals to a keyword
#2021-12-2214:57Ivan Fedorovok, I got it, I have to m/deref a [:ref :x] block, not :x itself#2021-12-2315:31Noah Bogart#re-frame has the function ->interceptor which has a parameter list of [& {:as m :keys [id before after]}] , which uses this new feature: https://clojure.org/news/2021/03/18/apis-serving-people-and-programs. how can i represent this in a malli function schema?#2021-12-2316:47Noah Bogarti have cobbled together this, but it feels pretty hacky, lol:
(m/=> ->interceptor
      [:=>
       [:cat
        [:or
         [:and
          [:map
           [:id qualified-keyword?]
           [:before {:optional true} fn?]
           [:after {:optional true} fn?]]
          [:fn (fn [{:keys [before after]}] (or before after))]]
         [:and [:catn
                [:id [:cat [:= :id] qualified-keyword?]]
                [:before [:? [:cat [:= :before] fn?]]]
                [:after [:? [:cat [:= :after] fn?]]]]
          [:fn (fn [[_ & args]] (pos? (count args)))]]]]
       :any])
#2021-12-2316:48Noah Bogartis there a way to make generative testing work with defn schemas like this? (mi/check) reports there’s no generator attached to this schema#2021-12-2316:51Noah Bogart
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.generator/no-generator {:options {:malli.core/function-checker #function[malli.generator/function-checker], :malli.generator/recursion {[:and [:map [:id qualified-keyword?] [:before {:optional true} fn?] [:after {:optional true} fn?]] [:fn #function[executor.interceptor/eval70188/fn--70190]]] 1, [:map [:id qualified-keyword?] [:before {:optional true} fn?] [:after {:optional true} fn?]] 1}}, :schema fn?}
#2022-12-2716:55ikitommi@nbtheduke bumped into that myself. Need to add proper support for the Keyword argument functions now also accept maps. Could you write an issue of that?#2022-12-2718:49Noah Bogarthttps://github.com/metosin/malli/issues/605#2022-12-2716:55Noah BogartYep, I can open an issue for it#2022-12-2717:00ikitommiis there a full spec for clojure destructuring?#2022-12-2717:09ikitommiseems to be that need to add something like :every and the merge of :map and :map-of to get proper support for parsing Clojure syntax.#2022-12-2717:10ikitomminaive 30min parser for extended Plumatic Schama defn syntax:
(m/parse
 Bind
 '[a :- :int
   [b :- :int [c :- :string :as d :- [:vector :any]] & e :as f]
   & {:keys [g h] :as i} :- [:map [:g :int] [:h :int]]])
;{:args [{:arg {:sym a}
;         :- :-
;         :schema :int}
;        {:arg {:vec {:elems [{:arg {:sym b}
;                              :- :-
;                              :schema :int}
;                             {:arg {:vec {:elems [{:arg {:sym c}
;                                                   :- :-
;                                                   :schema :string}]
;                                          :rest nil
;                                          :as {:as :as
;                                               :sym d
;                                               :schema {:- :-
;                                                        :schema [:vector :any]}}}}}]
;                     :rest {:amp & :form {:arg {:sym e}}}
;                     :as {:as :as
;                          :sym f
;                          :schema nil}}}}]
; :rest {:amp &
;        :arg {:arg {:map {:keys [g h] :as i}}
;              :- :-
;              :schema [:map [:g :int] [:h :int]]}}}
#2022-12-2717:10ikitommi
(def Bind
  (m/schema
   [:schema
    {:registry
     {"Schema" any?
      "Amp" [:= '&]
      "As" [:= :as]
      "Local" [:and symbol? [:not "Amp"]]
      "Map" [:map
             [:keys {:optional true} [:vector ident?]]
             [:strs {:optional true} [:vector ident?]]
             [:syms {:optional true} [:vector ident?]]
             [:or {:optional true} [:map-of simple-symbol? any?]]
             [:as {:optional true} "Local"]
             [:- {:optional true} "Schema"]]
      "Vector" [:catn
                [:elems [:* "SchematizedArgument"]]
                [:rest [:? [:catn
                            [:amp "Amp"]
                            [:form "SchematizedArgument"]]]]
                [:as [:? [:catn
                          [:as "As"]
                          [:sym "Local"]
                          [:schema [:? [:catn
                                        [:- "Separator"]
                                        [:schema "Schema"]]]]]]]]
      "Argument" [:alt
                  [:catn [:sym "Local"]]
                  [:catn [:map "Map"]]
                  [:catn [:vec [:schema [:ref "Vector"]]]]]
      "Separator" [:= :-]
      "SchematizedArgument" [:alt
                             [:catn
                              [:arg "Argument"]]
                             [:catn
                              [:arg "Argument"]
                              [:- "Separator"]
                              [:schema "Schema"]]]
      "Bind" [:catn
              [:args [:* "SchematizedArgument"]]
              [:rest [:? [:catn
                          [:amp "Amp"]
                          [:arg "SchematizedArgument"]]]]]}}
    "Bind"]))
#2022-12-2814:53ikitommiinferring function schemas, what would be better interpretation for these arguments:
'[a [b c]]
a) correct:
[:cat
 :any
 [:maybe
  [:cat
   [:? :any]
   [:? :any]
   [:* :any]]]]
b) strict:
[:cat
 :any
 [:schema 
  [:cat
   :any
   :any]]]
I would like to do b, but it’s Clojure and a is what the sequential destructuring does. let the user decide, default to a but allow a`strict` mode?
#2022-12-2814:59ikitommi
(defn kikka [a [b c]] [a b c])

(infer #'kikka)
;[:=>
; [:cat
;  :any
;  [:maybe
;   [:cat
;    [:? :any]
;    [:? :any]
;    [:* :any]]]]
; :any]
#2022-12-2814:59ikitommi
(defn ^:malli/strict strict-kikka [a [b c]] [a b c])

(infer #'strict-kikka)
;[:=>
; [:cat
;  :any
;  [:schema
;   [:cat
;    :any
;    :any]]]
; :any]
#2022-12-2815:01ikitommiwell, not doing the strict-mode now, but would be possible & easy#2022-12-2815:14Ben SlessWhat I'm missing most here is future support for named (or captured) type variables#2022-12-2816:55ikitommiwhat are named (or captured) type variables?#2022-12-2816:56ikitommi:cat could be :catn here, just need a good naming scheme for the unnamed things.#2022-12-2816:57ikitommialso, the whole thing is for full support of the plumatic syntax, so one can inline schemas.#2022-12-2816:58ikitommi
(ms/defn kikka :- [:vector :int] 
  [a :- :int [b :- :int, c :- :int]] 
  [a b c])
#2022-12-2816:59Ben SlessThe type if a signature [a] which is any, but an association that constrains the captured name for a in the body of the expression. This lays the groundwork for type inference engines later. Then you can unify the knowledge about that association with the environment (call site) and body. Essentially, introduce type variables / environment#2022-12-2817:00ikitommioh, that. I’m not doing that, hopefully smarter people will.#2022-12-2817:01Ben Sless😅#2022-12-2817:05Ben SlessI want to dive into it but I'm afraid I won't get anything#2022-12-2909:53ikitommiquestion: a new schema for "map or a sequence of map entries" or just an new property to :map schema? Relates to https://clojure.org/news/2021/03/18/apis-serving-people-and-programs.#2022-12-2909:55ikitommia) (m/validate [:map-like [:a :int]] [[:a 1]]) ;=> true b) (m/validate [:map {:coerce true}: [:a :int]] [[:a 1]]) ;=> true#2022-12-2909:57ikitommiI'm currently thinking of going with a, as this is a special case. Both ways, it's just few lines of extra code I think#2022-12-2909:58ikitommialso, what would be a good name for a new schema type, :map-like ? :every ?#2022-12-2909:59ikitommido you @nbtheduke the opinion for this?#2022-12-2909:59Ben Sless:entries#2022-12-2910:00Ben SlessThen :map is :entries + map?#2022-12-2910:03ikitommispec seems to have s/keys* for this#2022-12-2910:48ikitommithing is, transformers, parsers and explainers need separate code if we want to retain the original entry sequence. forcing the data to be a map allows us to reuse the current code.#2022-12-2910:49ikitommifor the Keyword argument functions now also accept maps thing, we just need a map, so a simple new way to coerce the entry sequence into a map would do.#2022-12-2913:04Noah BogartOne thing to note, the input isn't a sequence of map entries, but a sequence of alternating keys and values, which can be in any order #2022-12-2913:29ikitommiyes. I guess this is correct:
(require '[malli.destructure :as md])
(require '[malli.generator :as mg])

(defmethod mg/-schema-generator :any [_ _] (mg/generator :string))

(-> '[& {:as m :keys [id before after]}]
    (md/parse)
    :schema
    (mg/sample {:size 10, :seed 42}))
;(({:after ""})
; (:id "F")
; ()
; ({:after ""})
; (:before "n" :after "ai2" :id "5FI0" :id "")
; ({:after "qP3t"})
; ()
; ({:id "j"})
; ()
; (:before
;  "2"
;  :after
;  "HW5Mn3"
;  :after
;  "gv9m93"
;  :before
;  "GVDI2b"
;  :before
;  "fhR"
;  :before
;  "F8562"
;  :after
;  "lS"
;  :before
;  "Z7y0nz"
;  :before
;  "7G"))
#2022-12-2913:31ikitommiWIP https://github.com/metosin/malli/pull/606#2022-12-2915:23Noah Bogartthis matches the implementation, as far as i can tell! very cool#2022-12-2915:32Noah Bogartlooks like you’re missing this from the test suite, i’m not sure if you’re missing it from the schema:
user=> (defn example [{:keys [id before after] :as m}] [id before after m])
#'user/example
user=> (example '(:id 1 :before 2 :after 3 :missing 4))
[1 2 3 {:id 1, :before 2, :after 3, :missing 4}]
#2022-12-2915:34Noah Bogartwhich is to say, the bind is a destructured map, but the input can be a list of key-value pairs#2022-12-2915:36Noah Bogartweird thing, it can’t be a vector:
user=> (example [:id 1 :before 2 :after 3 :missing 4])
[nil nil nil [:id 1 :before 2 :after 3 :missing 4]]
#2022-12-2915:36ikitommiyou can pass in it as a list 🤯 by default???#2022-12-2915:38Noah Bogarthah yeah, it’s not really talked about anywhere, but the current version of clojure.core/destructure allows it: https://github.com/clojure/clojure/blob/clojure-1.11.0-alpha3/src/clj/clojure/core.clj#L4434-L4439#2022-12-2918:21ikitommi@nbtheduke, added support for it too. starts to smell like a new :map-destructuring Schema, which would hide the sequential part. Ugly, but works(?):
(-> '[a {:keys [b c]
         :strs [d e]
         :syms [f g]
         :or {b 0, d 0, f 0} :as map}]
    (md/parse)
    :schema)
;[:cat
; :any
; [:altn
;  [:map [:map
;         [:b {:optional true} :any]
;         [:c {:optional true} :any]
;         ["d" {:optional true} :any]
;         ["e" {:optional true} :any]
;         ['f {:optional true} :any]
;         ['g {:optional true} :any]]]
;  [:args [:schema
;          [:*
;           [:alt
;            [:cat [:= :b] :any]
;            [:cat [:= :c] :any]
;            [:cat [:= "d"] :any]
;            [:cat [:= "e"] :any]
;            [:cat [:= 'f] :any]
;            [:cat [:= 'g] :any]
;            [:cat :any :any]]]]]]]
#2022-12-2918:21Noah Bogartamazing. thanks so much for tackling this. destructuring in clojure is really weird haha#2022-12-2918:24ikitommi
(def Schema
  (-> '[a {:keys [b c]
           :strs [d e]
           :syms [f g]
           :or {b 0, d 0, f 0} :as map}]
      (md/parse)
      :schema))

(m/parse Schema [1 {:b 1, 'f 3, "e" 2, :extra 42}])
; => [1 [:map {:b 1, f 3, "e" 2, :extra 42}]]

(m/parse Schema [1 '(:c 1, , f 3, "e" 2, :extra 42)])
; => [1 [:args [[:c 1] [f 3] ["e" 2] [:extra 42]]]]
#2022-12-2914:49ikitommia thought experiment, should we have more argument relationship markers, e.g. :- (is a) and :< (a subset of)? could also go EXTREME EVIL and introduce real math symbols like :⊂ 👿
(def User 
  [:map 
   [:id :uuid] 
   [:name :string] 
   [:age :int]])

;; argument is exactly User
[{:keys [id age]} :- User]
; => [:cat [:map [:id :uuid] [:name :string] [:age :int]]]

;; argument should be subset of user (mark others as optional)
[{:keys [id age]} :< User]
; => [:cat [:map [:id :uuid] [:name {:optional true} :string] [:age :int]]]
#2022-12-2914:51ikitommialso, would be awesome if tools like #cursive and #calva would have special markers for the type hints, e.g. dim them out so it’s easier to read.#2022-12-2914:54ikitommi… for fully qualified keys, the key definitions could be pulled from the registry:
(mm/def ::id :uuid)
(mm/def ::name :string)
(mm/def ::age :int)

;; argument is exactly User
[{::keys [id age]}]
; => [:cat [:map ::id ::age]]
#2022-12-2914:54dharriganWe use (still 😞 ) compojure-sweet for some of our APIs, and I see a lot of :- in the path-params, body.... etc...and well, personally, I found it hard to know what :- meant etc...#2022-12-2914:56ikitommiin compojure-api, there is a lot of extra syntax, also the fnk syntax. sorry for all that 🙂#2022-12-2914:56dharriganI have to deal with it every day 😞 I definitely see lessons where learnt with the new improved reitit library 🙂#2022-12-2914:57dharriganSo, I'm not for, nor against, additional markers, I'm rather on the fence (just more stuff to learn I suppose!)#2022-12-2914:59ikitommiyeah, any support for inline typehints is a compromise. If IDEs would support that properly and there would be just one way to doing those, would be great.#2022-12-2915:19Noah Bogarti think i’d prefer words instead of single characters for such things. :exact or :exactly or :is-a are easier to parse than :- in my opinion. subset feels like Typescript-style structural typing, which clojure supports out of the box with open maps#2022-12-2915:20dharriganI'm with Noah on this one too, I would have to translate :- into is-a in my head as well. Nothing wrong in being wordy, when it comes to comprehension.#2022-12-2915:20dharriganmy 2c`s 🪙#2022-12-2915:25ikitommi#2022-12-2915:26ikitommi^:-- that’s what I would love to have, to remove the type/schema clutter if used.#2022-12-2915:30Noah Bogartah, i understand now that :- is supposed to be like static type declarations, that makes more sense#2022-12-2916:19Yehonathan SharvitIs there a commonly agreed way to document the meaning of each value in a :enum?#2022-12-2918:18ikitommi@viebel don’t think there is. ideas welcome#2022-12-2918:58Ben Slessenumn Where the keys are the enumeration and values are properties#2022-12-3007:47Yehonathan SharvitI don't get what you mean @UK0810AQ2#2022-12-3007:55Ben SlessLike the map syntax, where the keys are the enumeration and optionally you could provide a properties map#2022-12-3008:47ikitommi:enumn (or similar) would be good for key->value pair mappings.#2022-12-3008:48ikitommi
(def MyEnum
  [:enumn
   [:small {:description "so small"} "small"]
   [:medium {:description "such medium"} "medium"]])
#2022-12-3008:49ikitommi
(m/validate MyEnum "small") ; => true
#2022-12-3008:50ikitommi
(m/parse MyEnum "small") ; => [:small "small"]
#2022-12-3008:50ikitommi:thinking_face:#2022-12-3008:50ikitommiawesome#2022-12-3008:51ikitommie.g. :cat + :catn, :or + : orn, …#2022-01-0208:18Yehonathan SharvitI like the idea of enumn . I am wondering if there is any meaning to the keys (e.g. :small and :medium ).#2022-01-0208:27Ben SlessI was thinking of a slightly different semantic, [:enumn [v1 {:doc "foo}] [v2] v3], where the keys are the enumerations, and the syntax can be value | [value ?properties]#2022-01-0505:34Yehonathan SharvitI prefer what you suggested @UK0810AQ2 as we don't have to duplicate the enum values#2022-01-1009:35Yehonathan SharvitOpened an issue about this feature request https://github.com/metosin/malli/issues/613#2022-12-3008:53ikitommiOne of the oldest (and annoying) issue is how to describe a map + map-of. Need to take a stab at it, here are the options: 1) ternary closed
[:map {:closed [:string :int]}
 [:x :int]
 [:y :int]] 
2) new extra-keys (or such)
[:map {:extra-keys [:string :int]}
 [:x :int]
 [:y :int]]
3) ::m/default (like in :multi)
[:map
 [:x :int]
 [:y :int]
 [::m/default [:map-of :string :int]]]
#2022-12-3008:56ikitommileaning on 3, because: • it’s coherent way to describe “default in case none of the defined keys matched” • it’s easy to remove or add the key, e.g. (mu/assoc MyMap ::m/default [:map-of :uuid MyMap]):map-of already supports key-decoding so things like :uuid keys just work oob#2022-12-3008:56ikitommicomments welcome, original issue here: https://github.com/metosin/malli/issues/43#2022-12-3009:58Ben Sless4: [:map ^:of [int? int?]]#2022-12-3011:16ikitommimetadata looks good when writing, not that much when reading / serializing.#2022-12-3011:22Ben SlessI think this example is confusing because the map of spec and the entries don't match. A property of of on the map makes the most sense imo#2022-12-3011:36ikitommi:of sounds ok. what do you mean by: > the map of spec and the entries don’t match.#2022-12-3011:38ikitommihow would a schema in the properties be reported in m/explain?#2022-12-3011:41Ben SlessIf its :string then it can't have a keyword key. It has to be a union and not a superset. Which is confusing#2022-12-3011:43Ben SlessBut I'm not sure map-of and map should be unified#2022-12-3011:47ikitommiI read it “it’s a map with keyword keys :x and :y, the rest of the keys should be :string -> :int. Same with plumatic:
{:x s/Int, :y s/Int, s/Str s/Int}
#2022-12-3011:48ikitommie.g. {:x 1, :y 2, "z" 3, "å" 4} is valid in it.#2022-12-3011:49Ben SlessAs a user who has to work and communicate with others via code, I wouldn't want this to be valid#2022-12-3011:50Ben SlessI'd prefer that the specific keys will be a subset of the key schema and values be a subset of the value schema, not that the entire map describe a union#2022-12-3011:51ikitommiI see you point, would not want to use that myself, but that’s what JSON Schema, Plumatic and many others have atm.#2022-12-3011:52ikitommifor the destructuring, I would like to describe the namespaced keys with that#2022-12-3011:55Ben SlessJson schema has no notion of keywords, though#2022-12-3011:58Ben SlessAnd I don't mind saying Plumatic made a mistake by allowing it#2022-12-3011:50ikitommi5) wrap the extra keys with something like :schema to mark it’s a schema, not a real key.
[:map
 [:x :int]
 [:y :int]
 [[:schema :string] :int]]
#2022-12-3011:51Ben SlessWouldn't it require a special case in parsing?#2022-12-3011:52ikitommiyes#2022-12-3011:54Ben SlessLeaving surprises to your future self?#2022-12-3011:56ikitommiyes, I don’t think there is a right answer to this, just compromises. Not a fan of those (the reason the issue has been open for so long)#2022-12-3011:52Ben SlessShould map-of and map schemas be unified? One describes nominal tuples, the other a set of tuples#2022-12-3011:53ikitommihow could/should they be unified?#2022-12-3011:53juhoteperi::m/default makes sense, but using :map-of together with that seems funny. In :map schema the items are key-value pairs, :map-of describes full map. Something like [::m/default [:map-entry :string :int]] or just [::m/default :string :int] or [::m/default [:string :int]] (just force the :map default entry to always have two items) could be cleaner.#2022-12-3011:55ikitommiproblem with [::m/default :string :int] is that when you ask for m/children of the map, you get funny results.#2022-12-3011:56juhoteperiMaybe the :map-of makes sense as there can be multiple "default" / extra keys#2022-12-3011:57juhoteperiMaybe using ::m/extra name would describe better that this is the schema for those keys that aren't directly defined in :map schema#2022-12-3011:58ikitommiIn JSON Schema -land:
{
  "type": "object",
  "properties": {
    "number": { "type": "number" },
    "street_name": { "type": "string" },
    "street_type": { "enum": ["Street", "Avenue", "Boulevard"] }
  },
  "additionalProperties": { "type": "string" }
}
#2022-12-3012:03Ben SlessIt reads to me like the specified properties and additional properties should have the same key type#2022-12-3012:03Ben SlessBut json is limited#2022-12-3012:08ikitommiyes, we do not have that limitation#2022-12-3012:09ikitommibut, the extra keys are just for the case none of the actual keys hit#2022-12-3012:09ikitommithere is also :patternedProperties and other silly things.#2022-12-3012:27Ben Sless😕#2022-12-3012:09ikitommithe root need for this NOW btw is how to describe the namespaced keys in the destructuring syntax, elegantly. e.g. what is the schema for the map here:
(ns demo)

(let [{:keys [a1] ::keys [a2], :kikka/keys [a3]
       :syms [b1] ::syms [b2] :kikka/syms [b3]
       :strs [c1]
       :or {a1 0} :as map}
      {:a1 1, ::a2 2, :kikka/a3 3
       'b1 4 'demo/b2 5, 'kikka/b3 6
       "c1" 5}]
  [a1 a2 a3 b1 b2 b3 c1])
; => [1 2 3 4 5 6 5]
#2022-12-3012:11ikitommithought that would be a good reason to add the “extra keys” here, but not sure if that is needed. could be just :multi with dispatch on key qualification :thinking_face:#2022-12-3012:43Ben SlessOr map of enum to something?#2022-12-3012:45Ben SlessThis brings me back to my question about a schema about entries#2022-12-3012:53Ben Sless
entry := qualified | simple | or | as
qualified := [ns/kind form]
simple := [kind form]
kind := keys | syms | strs
or := [:or map-of,,,]
as := [:as symbol]
#2022-01-0116:54ikitommiunexpected help for the existing stuff:
(defn -map-like [x]
  (or (map? x)
      (and (seqable? x)
           (every? (fn [e] (and (vector? e) (= 2 (count e)))) x))))

(defn -keys-syms-key [k] 
  (-> k name #{"keys" "syms"}))

(def MapLike
  (m/-collection-schema
   {:type :map-like
    :empty {}
    :pred -map-like}))

(m/parse
 [MapLike
  [:or
   [:tuple [:= :keys] [:vector ident?]]
   [:tuple [:= :strs] [:vector ident?]]
   [:tuple [:= :syms] [:vector ident?]]
   [:tuple [:= :or] [:map-of simple-symbol? any?]]
   [:tuple [:= :as] symbol?]
   [:tuple [:and :qualified-keyword [:fn -keys-syms-key]] [:vector ident?]]]]
 '{:keys [b]
   :strs [c]
   :syms [d]
   :demo/keys [e]
   :demo/syms [f]
   :or {b 0, d 0, f 0} :as map})
;{:keys [b]
; :strs [c]
; :syms [d]
; :demo/keys [e]
; :demo/syms [f]
; :or {b 0, d 0, f 0} :as map}
… not the most performant, but good for the destructuring case 🥳
#2022-01-0116:54ikitommiunexpected help for the existing stuff:
(defn -map-like [x]
  (or (map? x)
      (and (seqable? x)
           (every? (fn [e] (and (vector? e) (= 2 (count e)))) x))))

(defn -keys-syms-key [k] 
  (-> k name #{"keys" "syms"}))

(def MapLike
  (m/-collection-schema
   {:type :map-like
    :empty {}
    :pred -map-like}))

(m/parse
 [MapLike
  [:or
   [:tuple [:= :keys] [:vector ident?]]
   [:tuple [:= :strs] [:vector ident?]]
   [:tuple [:= :syms] [:vector ident?]]
   [:tuple [:= :or] [:map-of simple-symbol? any?]]
   [:tuple [:= :as] symbol?]
   [:tuple [:and :qualified-keyword [:fn -keys-syms-key]] [:vector ident?]]]]
 '{:keys [b]
   :strs [c]
   :syms [d]
   :demo/keys [e]
   :demo/syms [f]
   :or {b 0, d 0, f 0} :as map})
;{:keys [b]
; :strs [c]
; :syms [d]
; :demo/keys [e]
; :demo/syms [f]
; :or {b 0, d 0, f 0} :as map}
… not the most performant, but good for the destructuring case 🥳
#2022-12-3104:17SwapneilHow does one make a relation between the input and output of a function? For instance, if I had the function (defn dothings [x y] (/ x y)), I want to check that (zero? (mod x :ret)). In spec you can do this with the :fn option in fdef, but I can't find any way to use malli's :fn option this way in a :=> schema#2022-12-3107:45ikitommi@ss2961 currently, no. but would be easy to add. Please write an issue out of that.#2022-12-3107:47ikitommie.g. with map-syntax:
{:type :=>
 :input {:type :cat, :children [{:type :int}]}
 :output :int}
=>
{:type :=>
 :input {:type :cat, :children [{:type :int}]}
 :output :int
 :fn ...}  
not sure where that should be in the vector-syntax. optional third element?
#2022-12-3107:57Ben SlessProperties?#2022-12-3108:16ikitommiIn this case, I think it's the way to go. The :=>satisfies the SchemaAST so it can read & write them from there. Brilliant!#2022-12-3107:54ikitommiadded some options to destructuring parser:
(defn parse
  "Takes a destructuring bindings vector (arglist)
   and returns a map with keys:

   | key            | description |
   | ---------------|-------------|
   | `:raw-arglist` | the original arglist (can have type-hints)
   | `:arglist`     | simplified clojure arglist (no type-hints)
   | `:schema`      | extracted malli schema
   | `:parsed`      | full parse results

   Parsing can be configured using the following options:

   | key                    | description |
   | -----------------------|-------------|
   | `::md/inline-schemas`  | support plumatic-style inline schemas (true)
   | `::md/sequential-maps` | support sequential maps in non-rest position (true)
   | `::md/required-keys`   | are destructured keys required (false)
   | `::md/closed-maps`     | are destructured maps closed (false)
   | `::md/references`      | are schema references used (false)

   Examples:

      (require '[malli.destructure :as md])

      (-> '[a b & cs] (md/parse) :schema)
      ; => [:cat :any :any [:* :any]]

      (-> '[a :- :string, b & cs :- [:* :int]] (md/parse) :schema)
      ; => [:cat :string :any [:* :int]]"
#2022-12-3107:55ikitommiany comments to https://github.com/metosin/malli/pull/606? naming, defaults etc. will ship that… next year 🙂#2022-01-0215:01ikitommiUpdating the README: pulling out malli function schemas from normal clojure functions:
(require '[malli.destructure :as md])

(def infer (comp :schema md/parse))

(defn kikka
  ([a] [a])
  ([a b & cs] [a b cs]))

(->> #'kikka
     meta
     :arglists
     (map infer)
     (map (fn [s] [:=> s :any]))
     (into [:function]))
;[:function
; [:=> [:cat :any] :any] 
; [:=> [:cat :any :any [:* :any]]
#2022-01-0215:05ikitommicould add a helper to enable that with malli.dev so that one could auto-infer all/interesting Vars at dev-team to get pretty runtime errors + clj-kondo mappings for free too.#2022-01-0218:30ikitommimerged. Before jumping into the-next-thing-I-need, I’ll try to address all PRs. Really good stuff there 🙇#2022-01-0218:32Ben SlessRegarding the string char ranges, @U02AH3D0HEV had a great idea for putting it in a lookup array, great perf#2022-01-0218:35Ben SlessAlso experimented with emitting a datomic(like) schema from schema#2022-01-0319:44SwapneilHow do you make a spec for a :repeat or :sequential with a certain length? I tried adding :count and :length in the properties, but neither seems to have an effect. My current spec is [:repeat {:min 0 :max 9} :int]]#2022-01-0414:32ikitommitry:
(m/explain [:repeat {:min 2, :max 2} :int] [1 2])
; nil

(m/explain [:repeat {:min 2, :max 2} :int] [1])
;{:schema [:repeat {:min 2, :max 2} :int],
; :value [1],
; :errors ({:path [0]
;           :in [1]
;           :schema :int
;           :value nil
;           :type :malli.core/end-of-input})}
#2022-01-0414:23pinkfrogIs the map syntax the recommended way to go? https://github.com/metosin/malli#map-syntax#2022-01-0414:33ikitommivector-syntax is the way to go, but map should work too. > NOTE: Map Syntax / SchemaAST is considered as alpha and subject to change.#2022-01-0414:34ikitommi#2022-01-0414:35ikitommiwith cursive + Clojure Extras + clj-kondo. Looking good 🙂#2022-01-0414:37dharriganI am liking the metadata version#2022-01-0414:37dharriganto me, precisely what metadata is all about 🙂#2022-01-0415:13emccuei like it too, just not that dev tooling can’t pick it up without a manual refresh#2022-01-0415:15emccueand the m/=> version there doesn’t work because if you load the whole file you load the schema, which triggers wrapping the function, and then load the function which will be unwrapped#2022-01-0415:31Karol WójcikPlumatic style. Cant wait to see it released ;3#2022-01-0415:54ikitommiI think we can solve the dev-tooöing issues with first two, with polling and var-watching.#2022-01-0416:31Michael Gardnerany progress on the timing problem with metadata? https://clojurians.slack.com/archives/C03S1KBA2/p1628602192164200#2022-01-0416:41ikitommiI don't think so, but adding a queue with a small delay should do the trick here.#2022-01-0417:42Michael Gardnerseems tricky to deal with the case where a var is defined and then immediately used#2022-01-0418:18Ben Slessteaser - how does emitting malli schema fron jackson annotations sound? 😛#2022-01-0420:55emccuei swear i dislike actual physical people named jackson at this point just by association#2022-01-0507:22Ben SlessI always think of Daniel Jackson from Stargate which puts me in a more positive mindset#2022-01-0507:55Ben SlessFiled under "thanks I hate it" https://github.com/bsless/malli-jackson#2022-01-0509:48eskosCan/could malli be used to extract/subselect a structure from a larger data structure? What I’m trying to achieve is that my CLJS app has one huge app-db ratom and from that I’d like to subselect certain parts which gets stored persistently with as little pain and whacking as possible, and I sort of had a thought that hey, it’d be nice to define the subselect as malli schema so that I could validate that the subselect is even persistable, plus malli schema is way easier to document than whatever else I’ve been thinking as solution to this so far. From system integrity’s point of view this could also work quite nicely, as marking specific branches/leaves of the data structure as persistable would mean writing a proper validator for it -> I wouldn’t have to worry someone breaking the, uh, persistability. And of course this is pretty straightforward thing to document to team members, both current and those who’ll join the project later that hey, just update the persistent malli schema if your data needs to be persisted.#2022-01-0510:13ikitommiThere is mt/strip-extra-keys-transformer to "select" a sub-schema.#2022-01-0513:49eskosHmm, interesting. :thinking_face:#2022-01-0512:59lauriThe following code snippet works with malli version 0.6.2:
(def instant
  (m/-simple-schema
   {:type            'instant
    :pred            (partial instance? Instant)
    :type-properties {:error/message "should be instant"}}))

(def Foo
  [:schema
   {:registry
    {::xyz
     [:map [:baz instant]]}}
   ::xyz])

(mu/merge [:map [:foo string?]]
          [:map [:bar Bar]])
=>
[:map
 [:foo
  [:schema
   {:registry #:test-namespace{:xyz [:map [:baz instant]]}}
   :test-namespace/xyz]]
 [:bar string?]]
but fails since version 0.7.0 with:
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.core/invalid-schema {:schema instant}
How can i execute it with the newest version on malli?
#2022-01-0513:05lauriwhen the type of the :baz field is changed to string? then it also works with the latest version but i’d like to be able to use a custom type..#2022-01-0513:57lauriseems that the malli.core/schema function fails in the newest version:
(m/schema Foo)
Error printing return value (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
:malli.core/invalid-schema {:schema instant}
#2022-01-0610:21ikitommi@U082J3TME It was a regression that is should be fixed with https://github.com/metosin/malli/commit/abef62dc682af7e1f61b841e5fed4038a1ccb01c, not released yet#2022-01-0610:21ikitommicould you retest with latest code in master?#2022-01-0610:22ikitommie.g. property registries were written as forms, which was a bad idea as there is no way to re-construct non-registered schemas using just the m/type information.#2022-01-0612:14laurijust checked with master and it works well now. Thanks a lot!#2022-01-0520:53annapawlickaHey folks, I’m trying to figure out if there’s a way to merge fields inside of registry. I’ve tried utility functions, tried declarative :merge , they don’t work. Here’s a small example of what I’m trying to do:
(def schema-registry-sample
  {::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [:merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [:merge
                                                                         ::offer ;; <<<< I want to add required field based on a certain condition
                                                                         [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]

   :sales-order [:map
                 [:paymentAuthorization ::payment-authorization]
                 [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order-explainer-sample
  (m/explainer [:schema {:registry schema-registry-sample}
                ::sales-order]))
The example here doesn’t work
{:type clojure.lang.ExceptionInfo
   :message ":malli.core/invalid-schema {:schema :merge}"
   :data {:type :malli.core/invalid-schema, :message :malli.core/invalid-schema, :data {:schema :merge}}
   :at [malli.core$_fail_BANG_ invokeStatic "core.cljc" 137]}
Other merge approaches fail as well. Any ideas? Is this just not doable right now?
#2022-01-0616:30ikitommihi @U05087MJH! the :merge is not enabled by default. You can register it globally with:
(mr/set-default-registry!
 (mr/composite-registry
  (m/default-schemas) ;; malli defaults
  (mu/schemas))) ;; plus the utility schemas
#2022-01-0616:31ikitommiso, this works:
(ns demo
  (:require [malli.registry :as mr]
            [malli.core :as m]
            [malli.util :as mu]))

(mr/set-default-registry!
 (mr/composite-registry
  (m/default-schemas) ;; malli defaults
  (mu/schemas))) ;; plus the utility schemas

(def schema-registry-sample
  {::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [:merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [:merge
                                                                         ::offer                                                                  [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]
   ::offer [:map]
   ::sales-order [:map
                  [:paymentAuthorization ::payment-authorization]
                  [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order-explainer-sample
  (m/explainer [:schema {:registry schema-registry-sample}
                ::sales-order]))
#2022-01-0616:35ikitommiIf you want spec-like custom mutable registry, here’s the thing:
(ns demo
  (:require [malli.registry :as mr]
            [malli.core :as m]
            [malli.util :as mu]))

(def registry* (atom {}))

(defn register! [type ?schema]
  (swap! registry* assoc type ?schema))

(mr/set-default-registry!
 (mr/composite-registry
  (m/default-schemas)
  (mu/schemas)
  (mr/mutable-registry registry*)))

(register!
 ::payment-authorization
 [:map
  [:id :string]
  [:approvalNumber :string]
  [:authResultDescription :string]
  [:authorizationDate :string]])

(register!
 ::sales-order-line-item
 [:map
  [:id :string]
  [:authorization [:maybe
                   [:merge
                    ::payment-authorization
                    [:map
                     [:type :string]
                     [:activationCode :string]
                     [:productCode :string]]]]]
  [:offer {:optional true} [:multi {:dispatch :offerProgram}
                            ["INSTANTSAVINGS" [:merge
                                               ::offer
                                               [:map [:gtin :string]]]]
                            [::m/default ::offer]]]])

(register!
 ::offer
 [:map])

(register!
 ::sales-order
 [:map
  [:paymentAuthorization ::payment-authorization]
  [:salesOrderLineItem [:vector ::sales-order-line-item]]])

(def sales-order-explainer-sample
  (m/explainer ::sales-order))
#2022-01-0621:07annapawlickaThank you @U055NJ5CC! I just realized I posted incomplete example, but you made it work anyway 🙂#2022-01-0720:49annapawlicka@U055NJ5CC Any chance to merge the mu/schemasinto my schema? I’d like to make the registry in this ns immutable as we’ll be using a lot of different schemas and there’s always a risk some other ns will set the global registry to something else. Simple merging doesn’t seem to work.#2022-01-0820:19ikitommiI think it’s possible. Need few minutest to test that one out.#2022-01-0820:26ikitommihere’s one way to do it: https://gist.github.com/ikitommi/af280065df4cb16a17123558e41463d3#2022-01-0820:27ikitommie.g. create a common registry for the base-schemas (immutable), pass it as argument to m/schema , can still add local schemas on top of it.#2022-01-0820:28ikitommimost workers (validators, explainers, parsers, unparsers, generators) are cached to the instance, so first m/validate on the Schema instance will create the validator, next will use the cached one.#2022-01-0820:37ikitommihappy to give a tech talk / discussion about malli for you team you are interested. There are lot’s of options of doing things. Also curious on where are you using it.#2022-01-0908:49ikitommitwo more options (added to gist):
;;
;; 3) registering utility schema via local registry (requires latest code from MASTER)
;;

(def schema-registry-sample
  {:merge (mu/-merge)
   ::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [:merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [:merge
                                                                         ::offer [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]
   ::offer [:map]
   ::sales-order [:map
                  [:paymentAuthorization ::payment-authorization]
                  [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order
  (m/schema
   [:schema {:registry schema-registry-sample}
    ::sales-order]))
#2022-01-0908:49ikitommi
;;
;; 4) using local component directly
;;

(def Merge (mu/-merge))

(def schema-registry-sample
  {::payment-authorization [:map
                            [:id :string]
                            [:approvalNumber :string]
                            [:authResultDescription :string]
                            [:authorizationDate :string]]

   ::sales-order-line-item [:map
                            [:id :string]
                            [:authorization [:maybe
                                             [Merge
                                              ::payment-authorization ;; <<<< I want to merge this with the fields below
                                              [:map
                                               [:type :string]
                                               [:activationCode :string]
                                               [:productCode :string]]]]]
                            [:offer {:optional true} [:multi {:dispatch :offerProgram}
                                                      ["INSTANTSAVINGS" [Merge
                                                                         ::offer [:map [:gtin :string]]]]
                                                      [::m/default ::offer]]]]
   ::offer [:map]
   ::sales-order [:map
                  [:paymentAuthorization ::payment-authorization]
                  [:salesOrderLineItem [:vector ::sales-order-line-item]]]})

(def sales-order
  (m/schema
   [:schema {:registry schema-registry-sample}
    ::sales-order]))
#2022-01-1020:57annapawlickaThank you! It looks like there’s quite a few options. We might take you up on your tech talk offer but we need to gather a list of questions first 🙂 We’ve been slowly replacing Plumatic schema with malli. Plus we started adding malli to a recent project that had no Plumatic schema - we use it to validate payload from services we call, but also to validate data that we send in our requests to them. Generation of test data is there as well but I think validation is the biggest part. And since these are all different data sources that we validate, we’re trying to find the best way to organize it. We’ve just started with it in December IIRC, so it will possibly grow. It also helped us to identify how much of the payload we actually use (it’s a big project, different graphql resolvers using the same source of data in different ways), and helped us make a decision around choosing the right response version. @UEH4D93GS has been looking into it from a different project than I so he might have a different perspective.#2022-01-0619:17Ben SlessAfter opening this I noticed there's been a similar request in this channel in the past. Anyone did work on the subject? https://github.com/metosin/malli/issues/609#2022-01-0713:44PanelTrying to understand why transformers won't compose, in this ex, (mt/transformers {:decoders ...}) is compiled but not run when the second transformer (mt/key-transformer ...) is present. If I remove (mt/key-transformer ...) then (mt/transformers {:decoders ...}) run just fine. I'm guessing this is related to interceptor :before and :after clashing for some reason.
(ma/decode
 [:map ["ContactMethodCde"
        [:enum {:lkp {"pcme" "Email"
                    "pcmp" "Phone"
                    "pcmm" "Mail"}}
         "Mail" "Phone" "Email"]]]
 {"ContactMethodCde" "pcme"}
 (mt/transformer
  {:decoders {:enum {:compile (fn [schema _]
                              (let [m (:lkp (ma/properties schema))]
                                (fn [x]
                                  (prn x)
                                  (m x))))}}}
  (mt/key-transformer
   {:decode {"ContactMethodCde" :contact-method}})))
This work, notice the :leave interceptor on :map decoders.
(ma/decode
 [:map ["ContactMethodCde"
        [:enum {:lkp {"pcme" "Email"
                      "pcmp" "Phone"
                      "pcmm" "Mail"}}
         "Mail" "Phone" "Email"]]]
 {"ContactMethodCde" "pcme"}
 (mt/transformer
  {:decoders
   {:enum {:compile (fn [schema _]
                      (let [m (:lkp (ma/properties schema))]
                        {:enter(fn [x]
                                  (prn x)
                                  (m x))}))}
    :map {:leave (mt/-transform-map-keys {"ContactMethodCde" :contact-method})}}}))
It wont work when the decoder is set to :enter :map {:enter ...}
#2022-01-0715:54Grant HornerIs there an equivalent to spec-alpha2's select in malli? i.e., it takes a :map schema and makes everything optional other than the specified keys?#2022-01-0721:41Panelhttps://cljdoc.org/d/metosin/malli/0.7.5/api/malli.util#required-keys maybe ?#2022-01-0820:17ikitommisomething like this?
(require '[malli.core :as m])
(require '[malli.util :as mu])

(defn select [schema keys]
  (-> schema
      (mu/optional-keys)
      (mu/required-keys keys)))

(select
 [:schema {:registry {::name :string
                      ::title [:enum "boss" "not-boss"]
                      ::age :int
                      ::skills [:set [:enum "clj" "cljs"]]}}
  [:map ::name ::title ::age ::skills]]
 #{::name ::age})
;[:map
; [:user/name :user/name]
; [:user/title {:optional true} :user/title]
; [:user/age :user/age]
; [:user/skills {:optional true} :user/skills]]
#2022-01-0820:17ikitommishould be easy to make it recursive, I think there is an issue to add such a tool to malli.util#2022-01-0922:59Grant Horner@U055NJ5CC that's definitely simpler than my solution… thanks!#2022-07-1921:16Noah Bogartwhat about this?
(defn select
  [schema ks]
  (let [schema (m/schema schema)
        plain-keys (filter keyword? ks)
        map-keys (->> ks
                      (filter map?)
                      (apply merge)
                      (not-empty))
        req-keys (vec (concat plain-keys (keys map-keys)))
        changed-schema (-> (mu/optional-keys schema)
                           (mu/required-keys req-keys))]
    (if-let [paths map-keys]
      (reduce (fn [schema [path v]]
                (assert (vector? v) "Nested map vals must be vectors")
                (mu/update schema path select v))
              changed-schema
              paths)
      changed-schema)))
#2022-07-1921:18Noah Bogartworks like this:
(select
  (m/schema
    [:map
     [:name string?]
     [:title [:enum "boss" "not-boss"]]
     [:skills [:set [:enum "clj" "cljs"]]]
     [:base [:map
             [:this-one [:map
                         [:one uuid?]
                         [:two int?]]]
             [:not-this-one string?]]]])
        [:name {:base [{:this-one [:two]}]}])
; [:map
;  [:name string?]
;  [:title {:optional true} [:enum "boss" "not-boss"]]
;  [:skills {:optional true} [:set [:enum "clj" "cljs"]]]
;  [:base [:map
;          [:this-one [:map
;                      [:one {:optional true} uuid?]
;                      [:two int?]]]
;          [:not-this-one {:optional true} string?]]]]
#2022-01-0820:17ikitommisomething like this?
(require '[malli.core :as m])
(require '[malli.util :as mu])

(defn select [schema keys]
  (-> schema
      (mu/optional-keys)
      (mu/required-keys keys)))

(select
 [:schema {:registry {::name :string
                      ::title [:enum "boss" "not-boss"]
                      ::age :int
                      ::skills [:set [:enum "clj" "cljs"]]}}
  [:map ::name ::title ::age ::skills]]
 #{::name ::age})
;[:map
; [:user/name :user/name]
; [:user/title {:optional true} :user/title]
; [:user/age :user/age]
; [:user/skills {:optional true} :user/skills]]
#2022-07-1921:16Noah Bogartwhat about this?
(defn select
  [schema ks]
  (let [schema (m/schema schema)
        plain-keys (filter keyword? ks)
        map-keys (->> ks
                      (filter map?)
                      (apply merge)
                      (not-empty))
        req-keys (vec (concat plain-keys (keys map-keys)))
        changed-schema (-> (mu/optional-keys schema)
                           (mu/required-keys req-keys))]
    (if-let [paths map-keys]
      (reduce (fn [schema [path v]]
                (assert (vector? v) "Nested map vals must be vectors")
                (mu/update schema path select v))
              changed-schema
              paths)
      changed-schema)))
#2022-01-0820:43ikitommiGiven a Var:
(defn -instrument
  "Takes an instrumentation properties map and a function and returns a wrapped function,
   which will validate function arguments and return values based on the function schema
   definition. The following properties are used:

   | key       | description |
   | ----------|-------------|
   | `:schema` | function schema
   | `:scope`  | optional set of scope definitions, defaults to `#{:input :output}`
   | `:report` | optional side-effecting function of `key data -> any` to report problems, defaults to `m/-fail!`
   | `:gen`    | optional function of `schema -> schema -> value` to be invoked on the args to get the return value"
  ([props]
   (-instrument props nil nil))
  ([props f]
   (-instrument props f nil))
  ([{:keys [scope report gen] :or {scope #{:input :output}, report -fail!} :as props} f options]
   ...
a simple schema inferrer:
(md/infer #'m/-instrument {::md/sequential-maps false})
;[:function
; [:=> [:cat :any] :any]
; [:=> [:cat :any :any] :any]
; [:=>
;  [:cat [:map
;         [:scope {:optional true} :any]
;         [:report {:optional true} :any]
;         [:gen {:optional true} :any]] :any :any]
;  :any]]
#2022-01-0820:43ikitommialso can read the plumatic hints. Could add it to read also normal Clojure typehints.#2022-01-0820:48ikitommiit can parse both the source code and the :arglists meta (both are described as malli sequence schemas). As we can create clj-kondo hints too, now we can do Var -> Malli -> Clj-kondo for free as in 🍺#2022-01-0820:49ikitommiit also understands the new Clojure keyword-ags-as-maps thing and the maps-as-sequence.#2022-01-0820:51ikitommithis is the real schema for it:
(md/infer #'m/-instrument)
;[:function
; [:=> [:cat :any] :any]
; [:=> [:cat :any :any] :any]
; [:=>
;  [:cat
;   [:altn
;    [:map [:map 
;           [:scope {:optional true} :any] 
;           [:report {:optional true} :any] 
;           [:gen {:optional true} :any]]]
;    [:args
;     [:schema [:* [:alt 
;                   [:cat [:= :scope] :any] 
;                   [:cat [:= :report] :any] 
;                   [:cat [:= :gen] :any] 
;                   [:cat :any :any]]]]]]
;   :any
;   :any]
;  :any]]
#2022-01-0910:42ikitommithe plumatic syntax is merged in master, also better dev-tooling for CLJS (thanks to @danvingo!). Looking forward to test reports, will release soon/after.#2022-01-0913:58Noah BogartThat’s so exciting! #2022-01-0917:44metameI’m interested in using malli with clj-kondo for static type checking. We already have clj-kondo in our project, and I saw in the malli readme on how to generate configs. Wondering if there are any decent examples that would demonstrate how to organize this for a large production code base?#2022-01-0917:59Ben SlessStatic type checking in Clojure is a bit limited without type inference, which malli and kondo don't do. You'll have to specify everything you want checked. A sane middle ground is having a "schema" namespace where you specify the schema of data coming in or out of your code base, then validate at the edges of the system.#2022-01-0918:06borkdudeclj-kondo does have some type inference#2022-01-0918:08Ben Sless(some type-inference? clj-kondo) ;; => true How much type inference does it do, exactly?#2022-01-0918:16borkdudeit has return type inference and locals inference for example#2022-01-0918:16borkdude
$ clj-kondo --lint - <<< '(let [x :foo] (inc x))'
<stdin>:1:20: error: Expected: number, received: keyword.
#2022-01-0918:17borkdude
$ clj-kondo --lint - <<< '(let [x (inc 2)] (assoc x :foo :bar))'
<stdin>:1:25: error: Expected: associative collection or nil, received: number.
#2022-01-0918:20Ben SlessDo associative collections carry around type information about their keys?#2022-01-0918:23borkdudethere is some of this, don't remember exactly how much, but probably not a lot#2022-01-0918:23borkdudethere is one open issue by @U055NJ5CC about this, that I haven't gotten around to yet#2022-01-0919:22metamewe already use plumatic.schema for data validation at the edges. I’m primarily talking about this: https://github.com/metosin/malli#clj-kondo I can see how to define the function spec right after with m/=> .#2022-01-0919:23metameWhat I’m wondering is if there’s a best practice for emitting all of the configs for a project after defining the specs in each ns?#2022-01-0919:26metameAnd further than that, would this buy me anything when it comes to associative collections?#2022-01-0919:27metameBut even just using it for arg types and return types, it’s still a step up imo#2022-01-0919:30borkdude@U774Z4VJA you get things for free and it might improve over time if you complain enough in the #clj-kondo channel :)#2022-01-0919:31borkdudeassociative collections: if you specify a required key as input to a function and you pass a map literal, then clj-kondo will be able to see it#2022-01-0919:31borkdudebut the inference on associative collection needs more love#2022-01-0919:31metameha, ya. Maybe if work stay calm enough, I may even be able to contribute here and there 😉#2022-01-0919:32metameAny tips on emitting config for a whole project?#2022-01-0919:33borkdudefor this question I defer to the malli authors!#2022-01-0919:33metameofc thanks#2022-01-0919:34metametl;dr of above thread. One unanswered question: Any tips on emitting clj-kondo config for a whole project?#2022-01-1014:36ikitommiThere is no guide for that and as there are 3+ ways to declare the schemas for functions, no best practices yet I think. I put the schemas next to the functions (my favourite is the plumatic-syntax, so inlined) and have the ! in the user / dev namespace, so you can do that easily. It instruments all the functions from all loaded namespaces, so should work ok.#2022-01-1014:36ikitommiif not, please report 🙂 planning to mallify a large codebase, can comment after that how it works / doesn’t.#2022-01-1014:37ikitommiif you are doing that already, user experience report post & feedback most welcome#2022-01-1014:39metameMany thanks for the response! Will ! also emit the clj-kondo config @U055NJ5CC?#2022-01-1014:43ikitommiyes#2022-01-1014:44ikitommihttps://github.com/metosin/malli/blob/master/src/malli/dev.clj#L15-L36#2022-01-1014:44metameThat is 🔥#2022-01-1014:44metameThanks#2022-01-1014:44ikitommi👍#2022-01-1014:48metameI also like the plumatic syntax so very cool that was just added#2022-01-1009:19Jungwoo KimHi I’m thinking of using malli with my project and still investigating it. I have a simple question. What do :cat and :catn means? I want to know the full name of cat and catn to understand better . alt and altn too.#2022-01-1009:40Ben SlessCat is a con*cat*enation of sequences Alt is an alternation of sequences The n suffix is "named", like a named capture group in a regular expression#2022-01-1009:41Ben SlessI can elaborate if this is unclear#2022-01-1009:44Jungwoo KimThanks a lot. I understood!#2022-01-1102:07Jungwoo KimHi! I’m really into malli very recently. thanks! I have a question though. Here’s my example pants-schema.
(def pants-schema
  [:and
   [:map
    [:id int?]
    [:size {:optional true} [:maybe :int]]
    [:size-alphabet {:optional true} [:maybe [:enum "S" "M" "L"]]]]
   [:fn {:error/message "size and size alphabet should be nil-matched"}
    '(fn [{:keys [size size-alphabet]}]
       (or (and (nil? size) (nil? size-alphabet))
           (and (some? size) (some? size-alphabet))))]])
You can see my nil-matched fn schema. I’m trying to use the schema to validate for functions like below,
(defn do-something-with-pants
  {:malli/schema [:=> [:cat pants-schema] :nil]}
  [pants]
  (prn pants)) ; do something
and then, if i pass the value {:id 1 :size 90} into the do-something-with-pants , I want to extract the error like humanize results but I got the full error messages.
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:137).
; :malli.core/invalid-input {:input [:cat [:and [:map [:id int?] [:size [:maybe :int]] [:size-alphabet [:maybe [:enum "S" "M" "L"]]]] [:fn #:error{:message "size and size alphabet should be nil-matched"} (fn [{:keys [size size-alphabet]}] (or (and (nil? size) (nil? size-alphabet)) (and (some? size) (some? size-alphabet))))]]], :args [{:id 1, :size 90}], :schema [:=> [:cat [:and [:map [:id int?] [:size [:maybe :int]] [:size-alphabet [:maybe [:enum "S" "M" "L"]]]] [:fn #:error{:message "size and size alphabet should be nil-matched"} (fn [{:keys [size size-alphabet]}] (or (and (nil? size) (nil? size-alphabet)) (and (some? size) (some? size-alphabet))))]]] :nil]}
I know explain + humanize is very useful to see the readable results but as I feel like that’s good for development stage. In the web application, My needs are very important to me. How do I handle errors properly?
#2022-01-1211:33rovanionAny recommendation on dealing with time? I've got multiple fields that hold LocalDateTime and it's not clear to me how I should express that.#2022-01-1215:24ikitommiyou could c&p code from this https://github.com/metosin/malli/pull/545, or wait for the PR or just do something like:
(def LocalDateTime
  (m/-simple-schema
    {:type 'LocalDateTime
     :pred #(instance? LocalDateTime %)
     :type-properties {:error/message "invalid date-time"
                       :decode/string #(LocalDate/parse %)
                       :json-schema {:type "string"
                                     :format "date-time"}}))

(m/validate LocalDateTime (LocalDateTime/now)) ; => true
(didn’t run that, might not work, but like that anyways.
#2022-01-1816:55rovanionThank you!#2022-01-1212:15rovanionWhen walking a schema, what does the nil that appears as the second entry in all children represent?
(def temperaturmätning                                                                                                                                                                                        
  (malli/schema                                                                                                                                                                                               
   [:map                                                                                                                                                                                                      
    [:odlingsplats [string? {:min 1}]]                                                                                                                                                                        
    [:tidpunkt     string?]]))

(malli/walk
  temperaturmätning
  (fn [schema path children _]
    (println "Path: " path)
    (println "Type: " (malli/type schema))
    (print "Schema: ")
    (clojure.pprint/pprint schema)
    (clojure.pprint/pprint children)
    (case (malli/type schema)
      string?  [:input {:type "text"} (-> path last name)]
      :map             [:div.input-group children])
    ))

;; [:div.input-group                                                                                                                                                                                          
;;  [[:odlingsplats nil [:input {:type "text"} "odlingsplats"]]                                                                                                                                               
;;  [:tidpunkt nil [:input {:type "text"} "tidpunkt"]]]]   
#2022-01-1212:23ikitommientry properties, e.g. in`[:map [:x {:optional true} :int]]`#2022-01-1213:32rovanionAh, of course!#2022-01-1216:20Ben Sless@ikitommi since the time PR is back on the agenda, what do you think about tackling the protocols split first?#2022-01-1218:06ikitommiThanks for reminding. I’ve been thinking about it. Just commented on the issue.#2022-01-1306:15Mutasem HidmiHi guys, how are you? I am new to Clojure. I wanted to ask a question about humanizing the errors returned from Malli when used as coercion. I tried to do like below, but it didn't work.
(def custom-coercion (reitit.coercion.malli/create
               {:transformers {:body {:default reitit.coercion.malli/default-transformer-provider
                                      :formats {"application/json" reitit.coercion.malli/json-transformer-provider}}
                               :string {:default reitit.coercion.malli/string-transformer-provider}
                               :response {:default reitit.coercion.malli/default-transformer-provider}}
                :error-keys #{:type :coercion :in :schema :value :errors :humanized #_:transformed}
                :compile mu/closed-schema
                :encode-error (fn [error] {:errors (:humanized error)})
                :validate true
                :enabled true
                :strip-extra-keys true
                :default-values true
                :options nil}))
#2022-01-1306:27dharriganI have an example#2022-01-1306:28dharrigangive me a moment and I'll push it up to github#2022-01-1306:35Mutasem HidmiThank @dharrigan. I appreciate it.#2022-01-1306:39dharriganHere you go: #2022-01-1306:39dharriganIt's a very simple example, which may help you along the path#2022-01-1306:40dharriganIn particular, look at:#2022-01-1306:40dharrigan#2022-01-1306:40dharriganThen#2022-01-1306:40dharrigan#2022-01-1306:40dharriganAnd here is the malli spec for the API:#2022-01-1306:40dharrigan#2022-01-1306:40dharrigan.#2022-01-1306:42dharriganThe encode-error function can be certainly improved (it assumes only one error, there could be multiple). Treat it like a starting point 😉#2022-01-1306:42dharriganHere is an example of a failure response:#2022-01-1306:42dharrigan
❯ http :8080/api/foo      
HTTP/1.1 400 Bad Request
Transfer-Encoding: chunked

What is going on?? There is a 'missing required key'. I was expecting a value for '[:id]' but got 'null' instead!
#2022-01-1306:42dharriganand here is an example of a successful response:#2022-01-1306:43dharrigan
❯ http :8080/api/foo?id=10
HTTP/1.1 200 OK
Content-Length: 14
Content-Type: application/json;charset=utf-8

{
    "hello": "10"
}
#2022-01-1306:43dharrigan.#2022-01-1306:43Mutasem HidmiThank you. I am checking it now#2022-01-1308:48Mutasem HidmiThanks buddy. By the way, you can put also this :error-keys #{:type :coercion :in :schema :value :errors :humanized #_:transformed} in order to get a humanized key that you can use in the encode-error#2022-01-1416:04metameAnyone have an example of plumatic syntax in the wild (not in docs yet)? (also possible I’ve misunderstood the meaning of adding plumatic syntax) I see it in the Destructuring section of the README for fn type hints. EDIT: looking at the PR, I’m seeing the usage under the experimental ns. So basically we can use in-line plumatic syntax for in-line function types but not anything similar to plumatic syntax for a full schema def. For that we still need to use the vector or map syntax.#2022-01-1416:34ikitommino full schema syntax. That would be... interesting.#2022-01-1416:48metamethanks 🙇#2022-01-1421:05mynomotoIs there someplace where the type options are documented? E.g. [:qualified-keyword {:namespace :xxx}] works but [:qualified-namespace {:namespace :xxx}] doesn´t check the namespace.#2022-01-1820:52coltnzHi, is there a way to default a value even if the key is optional in the schema?#2022-01-1820:52coltnzI have a map which may or may not have a date entry. In the latter case I want to default it to now#2022-01-1820:52coltnz
(m/decode [:map
           [:date {:optional true}
            [:maybe [:string {:decode/string mt/-string->date :default (tick/now)}]]]]
          {}
          (mt/transformer mt/string-transformer mt/default-value-transformer))
#2022-01-1820:53coltnzreturns {} which doesn't surprise me.#2022-01-1823:20Noah Bogart#2022-01-1906:01ikitommi@coltnz no at the moment. But we could add an option to mt/default-value-transformer to fill those, here’s the code for it: https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L394. Issue & PR welcome.#2022-01-1907:06coltnzcool i'll do a PR#2022-01-1916:51rovanionAnyone ever had their data thrown away during coercion in reitit when using their own custom malli schema type? I've defined mine like:
(defn- -string-gen [{:keys [min max]}]
  (cond
    (and min (= min max)) (gen/fmap string/join (gen/vector gen/char min))
    (and min max)         (gen/fmap string/join (gen/vector gen/char min max))
    min                   (gen/fmap string/join (gen/vector gen/char min (* 2 min)))
    max                   (gen/fmap string/join (gen/vector gen/char 0 max))
    :else                 gen/string))

(def postgres-string
  "Postgres does not allow the null byte \0 in strings.                                               
                                                                                                      
  The predicate only forbids null bytes, but the generator only generates ascii characters. This      
  severely limits the range of test strings. The latter should be changed at some later point."
  (malli/-simple-schema
   (fn [opts _]
     {:type            :postgres/string
      :pred            #(and (string? %) (not (string/includes? \0 %)))
      :type-properties {:error/message       "Unable to decode postgres string"
                        :decode/string       (fn [x] (prn "We are in the decoder") x)
                        :json-schema/type    "string"
                        :json-schema/format  "string"
                        :json-schema/minimum 0
                        :gen/gen             (-string-gen opts)}})))
But when defining a schema with it like
(def kasse
  (malli/schema
   [:map
    [:odlingsplats                  [postgres-string {:foreign-key "odlingsplatser"}]]
    [:diameter_m                    int?]
    [:djup_m                        int?]]))
and then running a request through my router like
(tove.handler/app-routes {:request-method :post, :uri "/kassar/-/new" :form-params {:odlingsplats "En bra plats" :diameter_m "1" :djup_m "1"}})
the data is without any error conformed into
{:diameter_m 1, :djup_m 1} 
If I replace postgres-string with string? in the schema the data gets passed on and the conformed map looks like:
{:odlingsplats "En bra plats", :diameter_m 1, :djup_m 1}
#2022-01-1916:54Ben SlessHow did you define coercion for the router?#2022-01-1916:57rovanionHere's the relevant section of the router:
["/-/new"                     {:post {:handler (partial page/create-entry! "kassar" "id")
                                           :coercion mc/coercion
                                           :parameters {:form s/kasse-user-data}}}]]
I also have this at the very end of the router definition:
{:data {:middleware [rrc/coerce-request-middleware]}}
#2022-01-1917:03Ben SlessI think you need to specify it should use string coercion#2022-01-1917:32rovanionI'm not sure I understand, where should I tell what what?#2022-01-1917:33Ben SlessI'll dig up an example#2022-01-1917:34Ben SlessI think you need something like
(mcoercion/create
   {:transformers
    {:body
     {:default string-transformer-provider
      :formats {"application/json" string-transformer-provider}}
     :string
     {:default string-transformer-provider}
     :response
     {:default mcoercion/default-transformer-provider}}})
#2022-01-1917:34Ben SlessRoughly#2022-01-1917:35Ben Slessidea - use string coercion instead of json coercion by default for your params#2022-01-2009:22rovanionWhat seems to have worked is that I set up my own spec-style mutable registry that I set as the default registry. And by referring to the type as a key in kasse instead of as a var it all worked out.#2022-01-2009:23rovanionNow my "We are in the decoder" message is printed and all is fine!#2022-01-1919:55Noah BogartIs there a shorter form of
(def FxSchema
  (m/schema
    [:* [:or
         nil?
         [:tuple qualified-keyword?]
         [:tuple qualified-keyword? [:maybe any?]]]]))
#2022-01-1920:35Ben Slessor nil? T -> maybe T#2022-01-1920:36Ben Sless[:sequence qualified-keyword? [:? any?]]#2022-01-1920:37Ben Sless[:* [:maybe [:sequence qualified-keyword? [:? any?]]]]#2022-01-1920:37Noah Bogartvery nice, thank you#2022-01-1920:40Noah Bogarthm, that doesn’t work because :sequential requires homogenous input. maybe i need :cat?#2022-01-1920:41Noah Bogartthat’s it: :sequential -> :cat works#2022-01-1920:41Noah Bogartthanks for the help!#2022-01-1920:45Ben Sless👍#2022-01-1920:45Ben SlessWhen in doubt, grab a catjam#2022-01-2004:57Abhinavhow do I express this in a schema?
{"hi" {:greeting "hi" :id 1} "hello" {:greeting "hello" :id 2} "hiee" {:greeting "hiee" :id 3}}
I have a function that transforms a map like this
[{:greeting "hi" :id 1}{:greeting "hello" :id 2}{:greeting "hiee" :id 3}]
into the one above, the values of :greeting aren’t known before hand
#2022-01-2011:16ikitommimaybe: [:map-of :string [:map [:id :int] [:greeting :string]]]#2022-01-2011:23AbhinavThat is a good suggestion, but I want to say that the key of the map has to the be equal to the value of the :greeting key. so [:map-of :string [:map [:id :int] [:greeting :string]]] would be valid for {"foostring" {:greeting "hi" :id 3}} but the above won’t be valid#2022-01-2011:24AbhinavI don’t know if what I’m tryign to do is an anti-pattern:sweat_smile:#2022-01-2011:37ikitommino, it’s all valid, maybe:
(def Schema
  (m/schema
   [:and
    [:map-of :string [:map [:id :int] [:greeting :string]]]
    [:fn (partial every? (fn [[k d]] (= k (:greeting d))))]]))

(m/validate
 Schema
 {"hi" {:greeting "hi" :id 1}
  "hello" {:greeting "hello" :id 2}
  "hiee" {:greeting "hiee" :id 3}})
; => true

(m/validate
 Schema
 {"hi" {:greeting "<<invalid>>" :id 1}
  "hello" {:greeting "hello" :id 2}
  "hiee" {:greeting "hiee" :id 3}})
; => false
#2022-01-2011:39ikitommim/explain here doesn’t produce correct place of error, could should do it, we could tune either :fn to support returning custom explain-results, or add support for generic :explain key for keys, or something. Someone should write an issue out of this.#2022-01-2103:25Abhinav@U055NJ5CC thank you for your help. that was exactly what I was looking for.#2022-01-2012:37rovanionI believe I'm just being daft here but I haven't been able to figure this out for more than an hour so: How do I resolve a keyword to a schema in the registry?
(def registry
  (atom {}))

(defn register! [type ?schema]
  (swap! registry assoc type ?schema))

;; Combine the default registry with our own mutable registry.
(mreg/set-default-registry!
 (mreg/composite-registry
    (mreg/fast-registry (malli/default-schemas))
    (mreg/mutable-registry registry)))

(register! :db/kasse
   [:map
    [:id                            [:int {:primary-key true :db-generated true}]]
    [:odlingsplats                  [:string {:foreign-key "odlingsplatser"}]]
    [:diameter_m                    :int]
    [:djup_m                        :int]
    [:volym_m2                      [:int {:db-generated true}]]])

(malli/walk
 :db/kasse
 (malli/schema-walker identity))
;; => :db/kasse
I've tried wrapping :db/kasse in different functions from malli but none seem to do the lookup. The lookup function in the core namespace is private. Just running (:db/kasse malli/default-registry) does not work either. (malli/schema :db/kasse) seems like the obvious choice but it seemingly has no effect.
(malli/walk
 (malli/schema :db/kasse)
 (malli/schema-walker identity))
;; => :db/kasse
#2022-01-2012:58ikitommithe :db/kasse returned is a Malli Schema instance, it’s print output is just the form, so looks like keyword. It’s type is :malli.core/schema, which is the internal eager reference, like a Var in Clojure. If you want to get the schema behind it, you can m/deref it. But, calling m/validate on :db/kasse works too. the :malli.core/schema forwards the calls to the actual instance, like Var. Hope this helps.#2022-01-2012:59ikitommialso, walking it just work, try something like (m/ast :db/kasse) to verify#2022-01-2013:58rovanionThank you, dereffing worked perfectly. Though I'm not sure I understand the latter reply:
(let [schema (malli/schema :db/kasse)]                                                                                
  (prn (type schema))                                                                                                 
  (malli/walk                                                                                                         
   schema                                                                                                             
   (malli/schema-walker identity))) 
;; => :db/kasse
;; printed: :malli.core/schema
#2022-01-2016:49mafcocincohi all! Just started using malli and I have what I think is a pretty newbie question: Is there a way, in the context of a :map schema declaration to have mutually exclusive keys? That is “map should contain :a and :b together or :c by itself”.#2022-01-2017:39Ben SlessNot atm, but it's an often requested feature#2022-01-2018:21jeroenvandijkHi, I’ve been playing with the idea of using Malli as a way to define the main data model of an application. Many features are already in Malli I believe. One thing that might be missing is an easy way to generate consistent test and seed data. Maybe something like composite types are an useful addition?
(register! :user/first-name [:enum "john" "joe" "alex"])
(register! :user/last-name [:enum "smith" "wood" "lee"])

(register! :user/full-name
           [:composite {:schema string?
                        :compose (fn [first-name last-name]
                                   (str first-name " " last-name))
                        :fields [:user/first-name :user/last-name]}])

(register-entity! ::user
                  [:user/first-name
                   :user/last-name
                   :user/full-name])

(generate ::user)
;; => #:user{:first-name "joe", :last-name "wood", :full-name "joe wood"}
More here https://gist.github.com/jeroenvandijk/5e0785f25f7fdfeac7bc7a0be72cb62a#file-malli_composite_test-clj
#2022-01-2019:39Ben SlessDid you just implement unification?#2022-01-2019:56jeroenvandijkNot consciously 😅 Maybe accidentally? I was just thinking how do I get meaningful seed and test data when I’m just defining my attributes and entities#2022-01-2020:09jeroenvandijk@UK0810AQ2 I see you did some work on unification yourself (https://github.com/metosin/malli/issues/474 and https://github.com/bsless/malli-keys-relations). My approach is not that advanced (or complete)#2022-01-2020:14Ben SlessIt's okay, my approach sucked anyway I'm becoming convinced the only solution is embedding micro kanren and I'm scared#2022-01-2020:21jeroenvandijkHaha I can imagine#2022-01-2110:46Karol WójcikDoes malli supports inline schemas in map destructuring syntax in defn? e.g.,
(mx/defn kikka 
  [{:keys [
     hello :- :string
     world :- :string
  ]}]
  (println hello world))
#2022-01-2110:48Karol WójcikIt looks like not.
:schema
 [:=>
  [:cat
   [:altn
    [:map
     [:map
      [:hello {:optional true} :any]
      [:- {:optional true} :any]
      [:string {:optional true} :any]
      [:world {:optional true} :any]]]
    [:args
     [:schema
      [:*
       [:alt
        [:cat [:= :hello] :any]
        [:cat [:= :-] :any]
        [:cat [:= :string] :any]
        [:cat [:= :world] :any]
        [:cat :any :any]]]]]]]
  :any],
#2022-01-2113:06ikitommididn’t add that as the current support is “plain plumatic”.#2022-01-2113:06ikitommibut, plumatic doesn’t support different returns from arities, so we should extend it anyway#2022-01-2113:07ikitommibut, if that is supported, one can’t have a key named -, right?#2022-01-2113:14ikitommi
(let [{:keys [a :- :int]} {:a 1, :- 2, :int 3}] [a - int]) 
; => [1 2 3]
#2022-01-2113:16ikitommiso, don’t think it’s a good idea to support it, what do you think?#2022-01-2116:43Karol WójcikRight. After consideration I also think the syntax where the schema is specified in keys destructuring, is pretty hard to understand. #2022-01-2202:49jkrasnayIs there a way to get function instrumentation working in CLJS? There seems to be some stuff in malli.instrument.cljs but my CLJS build complains that that ns is not available.#2022-01-2207:30ikitommiHaven’t used that myself, but the tests pass. Are you depending on the latest version from master? It’s not released yet#2022-01-2215:22jkrasnayOh, I was on 0.7.5. Thanks!#2022-01-2220:19dvingo@U0DTSCAUU feel free to ping me if you run into any issues!#2022-01-2207:30ikitommiadded .pretty/explain#2022-01-2211:22ikitommifrom cursive#2022-01-2211:21ikitommipushed all current stuff for leiningen users as`[metosin/malli "0.8.0-SNAPSHOT"]`#2022-01-2213:36Karol Wójcik@ikitommi FYI: The CLJ Kondo config has unbalanced bracket 🧵#2022-01-2213:36Karol WójcikA PR with issue fixed https://github.com/metosin/malli/pull/625#2022-01-2214:32ikitommiThanks! Merged#2022-01-2215:28jkrasnayI’m having a problem with instrumenting my functions. I define a function in namespace A and register it with Malli as follows:
(m/=> selectable-label [:=> [:cat [:map
                                   [:build map?]
                                   [:selectable-id id/SelectableId]]]
                        locale/LocalizableString])
Then in namespace B I call
(mi/instrument!)
This triggers errors when compiling namespace B that namespaces id and locale are not found. These are aliases from the :require block in namespace A. Is there any way around this?
#2022-01-2215:42jkrasnayBTW namespace A is cljc while namespace B is cljs.#2022-01-2215:46jkrasnayIt works if I fully qualify id and locale.#2022-01-2216:26jkrasnayI think this may be one of those subtle things about CLJS macros, where we can’t qualify the namespace since the macro is running in CLJ.#2022-01-2216:58Игорь ЛисовцовHi there guys. I'm new to clojure and especially to malli. So, I'm playing around repl and malli, and notices that below example from the documentation doesn't work
(m/decode
  [string? {:decode {:string 'str/upper-case}}]
  "kerran" mt/string-transformer)
; => "KERRAN"
and instead of "KERRAN" it returns "kerran". Func at the place {:string func simply not called. Did I do something wrong?
#2022-01-2217:12ikitommiThe docs for this part are ahead of the release, plan is to release 0.8.0 n the next few days. If you know how to depend on the latest commit of malli, you can test that too.#2022-01-2217:18Игорь ЛисовцовGot it. Thank you a lot!#2022-01-2308:56pithylessI feel I'm missing something obvious; if I've got a [:map [:name :string] [:age :int]] schema, what's the best way to get back the coll of keys [:name :age]?#2022-01-2309:26pithylessLooking at the transformers, I guess it would be:
(some->> schema m/entries (map first) seq)
#2022-01-2312:05ikitommiyes, that's it.#2022-01-2316:03ikitommi🥳#2022-01-2422:23lsenjovThis is the real Christmas present I’ve been waiting on 🥳#2022-01-3001:39Grant HornerThis is awesome! I never used schema, and I'm loving the new malli.experimental/defn. How do you make it play nicely with Cursive? I've been unable configure the cursive resolution to get rid of all the yellow squigglies#2022-01-3007:22ikitommithanks! You should mark it to be resolved as schrma.core/defn. And go vote up https://github.com/cursive-ide/cursive/issues/2645#2022-01-2719:55Noah BogartI have a map that contains function values, but I want to use a Var (`#'function-name`) to facilitate repl-driven development, however fn? doesn’t consider the Var a function and malli doesn’t recognize var? as a schema function#2022-01-2720:14Ben SlessYou want to validate the map itself?#2022-01-2720:15Noah BogartI have the validation for the map, but i’m realizing I need to be able to say [:or fn? var?] instead of just fn?#2022-01-2720:15Ben SlessYou can use a decoder#2022-01-2720:16Ben SlessAnd deref the values :)#2022-01-2720:17Ben SlessI.e. [:map [:k [fn? {:decode/var deref}]]]#2022-01-2720:17Ben SlessApproximately#2022-01-2720:17Noah Bogarthuh interesting. i’ll give that a shot! thanks#2022-01-2720:19Ben SlessCheers. Make sure to wrap that deref in a try catch#2022-01-2720:19Ben SlessBut, why are you validating a map of functions? Especially w/o function schema it isn't very informative, is it?#2022-01-3118:07Noah Bogartjust barebones checking that i’m creating the right types, not doing much more than that to start#2022-01-2719:56Noah Bogartany way to handle this?#2022-01-3117:55Brett Rowberrywhen inferring malli schemas from data, I can tell it how to handle strings as uuids, for example, as shown here https://github.com/metosin/malli/pull/597 some string values may be inst or other types - is there a way to have it handle those as well?#2022-01-3118:15ikitommisure, anything you can present transformers works here. Check how inst? is handled in malli.transform#2022-01-3119:16ikitommihttps://github.com/metosin/malli/pull/632#2022-02-0117:53Brett Rowberrysorry, I should have given a better example
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])

(mp/provide
 [{:my-uuid "caa71a26-5fe1-11ec-bf63-0242ac130002"
   :my-inst "2021-01-01T00:00:00Z"}]
 {::mp/value-decoders {'string? {:uuid mt/-string->uuid}}})
;; can I treat :my-uuid as a uuid and :my-inst as an inst?
#2022-02-0118:55ikitommiwith master:
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])

(mp/provide
 [{:my-uuid "caa71a26-5fe1-11ec-bf63-0242ac130002"
   :my-inst "2021-01-01T00:00:00Z"}]
 {::mp/value-decoders {'string? {:uuid mt/-string->uuid
                                 'inst? mt/-string->date}}})
; => [:map [:my-uuid :uuid] [:my-inst inst?]]
#2022-02-0121:46Brett Rowberrythank you!#2022-02-0123:00Brett Rowberryso this would be after 0.8.0#2022-02-0207:41ikitommiif you use deps, you can depend on the lastest commit on master to try this out. for leiningen - in the next release.#2022-02-0214:11Brett Rowberrywe're using some custom bazel thing, so I'll have to wait for the jar. thank you so much for this feature! type providers were so awesome in F# - I really appreciate having something similar in Clojure!#2022-02-0513:01ikitommi[metosin/malli "0.8.1"]#2022-02-0513:01ikitommiand good to hear, it is useful 🙂#2022-02-0118:55ikitommiwith master:
(require '[malli.provider :as mp])
(require '[malli.transform :as mt])

(mp/provide
 [{:my-uuid "caa71a26-5fe1-11ec-bf63-0242ac130002"
   :my-inst "2021-01-01T00:00:00Z"}]
 {::mp/value-decoders {'string? {:uuid mt/-string->uuid
                                 'inst? mt/-string->date}}})
; => [:map [:my-uuid :uuid] [:my-inst inst?]]
#2022-02-0117:50Grant HornerIs there a way to truncate the output of the pretty reporter? I’m getting occasional reports for functions that take in vectors of 10,000+ elements and the output is absolutely crushing my poor intellij repl#2022-02-0118:53ikitommilooking at fipp docs, try setting clojure.core/*print-length* & clojure.core/*print-level*, https://github.com/brandonbloom/fipp#printer-usage.#2022-02-0118:53ikitommiif that works, doc PR to malli welcome#2022-02-0217:23Grant HornerJust did! I appreciate the quick response, I could’ve figured it out if I had read the documentation more in-depthly :face_palm:#2022-02-0217:27ikitommiThanks! Merged#2022-02-0314:32lreadJust noticed this in my vid feed: https://youtu.be/mNpE7cPm-N8#2022-02-0408:57JacquesIs it possible to use registry references for function schema input parameters? Is there an example somewhere to see how this might work?#2022-02-0513:16ikitommi@jacquesdpz maybe:
(require '[malli.core :as m])
(require '[malli.registry :as mr])

(mr/set-default-registry!
 (mr/composite-registry
  (m/default-schemas)
  {::user-id :int
   ::user [:map
           ::user-id
           [:name :string]
           [:age :int]]}))

(require '[malli.experimental :as mx])
(require '[malli.generator :as mg])

(mx/defn get-user :- ::user [id :- ::user-id]
  (assoc (mg/generate ::user) ::user-id id))

(get-user 123)
; => {:kikka/user-id 123, :name "8p28ySz", :age 2405534}
#2022-02-0709:13Jacquesthank you @U055NJ5CC#2022-02-0604:23pinkfrogIs it possible to specify the :min/:max attributes for regular expressions (instead of doing it inside the re pattern)?#2022-02-0608:00ikitommino. Regal (https://github.com/lambdaisland/regal) has malli-support, but not sure it's up-to-date with the latest version.#2022-02-0714:44Ben SlessWhere does it make sense in your opinion to specify keywords renaming for a transformer? in a map schema's property or on its entry?#2022-02-0714:45Ben Sless[:map {:rename {:a :b}} [:b int?]] vs [:map [:b {:rename :a} int?]]#2022-02-0716:40ikitommiMy intuition says it belongs to the map. But I guess, it depends. Being part of the entry-tuple, you could for example merge two maps with mapping to same domain and the mappings would be merged automatically#2022-02-0716:41ikitommie.g.
[:merge
 [:map [:a {:rename/SAP "SAP_A", :rename/SALESFORCE "sf_a"} int?]]
 [:map [:b {:rename/SAP "SAP_B"} int?]]]
#2022-02-0716:42ikitomminot actually sure if the entry properties get merged correctly here, just guessing 🙂#2022-02-0717:54Ben SlessFollowup harder question - if I rename in decode it reports error on the wrong key path!#2022-02-0717:54Ben SlessI dug myself a deeper hole than expected 😄#2022-02-0717:54Ben SlessHow do I get out?#2022-02-0806:27Ben SlessThe only way I see around it is attaching as metadata the rename mappings and encode the error report on response#2022-02-0818:01Ben SlessBump / help?#2022-02-0819:00ikitommicould you gist a minimal case, I can try to figure out howto#2022-02-0819:05Ben SlessSure, I'll send it over later today / tomorrow morning#2022-02-0909:05Ben Sless
(defn key-renamer
  "Take a tuple of keys [`k1` `k2`] and return a function of a map `m`
  which will replace `k1` with `k2` if it exists in `m`"
  [[k1 k2]]
  (fn -rename [m]
    (if-let [[_ v] (find m k1)]
      (dissoc (assoc m k2 v) k1)
      m)))

(defn- -compile-rename-keys-transformer
  "Takes a map of keys->keys and returns a function which will rename the
  keys in LHS to RHS if they exist in a map."
  [m]
  (if (seq m)
    (reduce comp (map key-renamer m))
    identity))

(def rename-keys-transformer
  (mt/transformer
   {:decoders
    {:map
     {:compile
      (fn [schema _]
        (-> schema m/properties :rename -compile-rename-keys-transformer))}}}))

(comment
  (m/decode
   [:map
    {:rename {:a :b :c :d :x :y}}
    [:b int?]
    [:c int?]
    [:y int?]]
   {:a 1
    :c 2
    :x 3}
   (mt/transformer rename-keys-transformer)))
Given that an input is incorrect post transformation, how do I report the original field name as invalid?
#2022-02-0909:05Ben Sless@U055NJ5CC#2022-02-1110:05ikitommihad time to look at this. Thing is, you should describe the resulting schema, not the orginal.#2022-02-1110:06ikitommiso, it should be:
[:map 
 {:rename {:a :b, :c :d, :x :y}}
 [:b int?]
 [:d int?]
 [:y int?]]
#2022-02-1110:07Ben SlessThe problem is that schemas, being a description of "ought", can't report back errors about "is" for any lossy transformation#2022-02-1110:08Ben SlessI had a typo in the last entry which I can spot now#2022-02-1110:10Ben SlessAnyway, schema validates b, user gave me a, I want to report back "a should be an integer" if I was given a, even thought I treat it as b. Some sort of alias mechanism#2022-02-1110:10ikitommiok, that :thinking_face:#2022-02-1110:11ikitommiwhat if the rename-map is used to create the target (or source) schema?#2022-02-1110:11ikitommie.g. transform keys.#2022-02-1110:11ikitommithen, you could validate them separately.#2022-02-1110:18ikitommi
(def Target
  (m/schema
   [:map
    [:b int?]
    [:d int?]
    [:y int?]]))

(def mappings {:b :a, :d :c, :y :x})

(def Source
  (mu/transform-entries 
   Target 
   (partial map (fn [[k p v]] [(mappings k k) p v]))))

Source
; => [:map [:a int?] [:c int?] [:x int?]]
#2022-02-1110:19ikitommisomething like that, explicit schemas for both Source and Target.#2022-02-1110:20ikitommiI would like to see someone cook Meander and Malli together, so that one could write a Malli schema and a Meander transformation for it and infer the target Malli schema from those.#2022-02-1110:20ikitommimight be the ultimate data transformation tool 😎#2022-02-1110:21Ben SlessI started off with a separate schema, I wanted to be clever and unify#2022-02-1110:22Ben SlessRegarding meander, I also thought on using it for the complex unification problems I made for myself#2022-02-1110:23Ben SlessAdd a "match" property to schema which binds, then compile that to meander pattern and match on it#2022-02-1110:24Ben SlessOr going the other way, write a meander to malli compiler#2022-02-1110:24Ben SlessThen schemas reflect how data looks#2022-02-1110:25Ben SlessAnd you can go crazy with unification#2022-02-0716:09AJ JaroWhat is the best way to setup the schema to work with a protocol? We could potentially use the https://github.com/metosin/malli#fn-schemas to work through instance?, but maybe there’s a better solution#2022-02-0716:31ikitommiSee https://github.com/metosin/malli/issues/555#2022-02-0815:00AJ JaroThanks. We’ll probably implement some fn schema to check for satisfies or instance for now I guess!#2022-02-0720:20Ben Wadsworth#2022-02-0720:27Ben Wadsworthdoh’ i mean the obvious here is wrong…one sec..#2022-02-0720:28Ben Wadsworth(being the map in one call and string in another) lol…#2022-02-0720:35Ben WadsworthI compounded a couple of issues but I dont think its Malli…. I think it might be coercion from reitit. Sorry bout that, another pair of eyes found the painfully obvious string where a map should have been in my example 😐#2022-02-0721:04Ben Wadsworthyeah no issue… happy monday#2022-02-0818:01Ben SlessBump / help?#2022-02-0811:10Mutasem HidmiHow can set field to be not required in malli Coercion#2022-02-0813:28dvingodoes {:optional true} not work?#2022-02-0814:39pinkfrogI am seeing this coercion failure, but dunno why it fails. It claims missing key, however the key is there.#2022-02-0814:39pinkfrog
{
  "schema": "[:map {:closed true} [:email [:re #\"^
#2022-02-0815:19dvingovalue has email as string not a keyword#2022-02-0815:37pinkfrogYes.#2022-02-0817:58CaseyIs it possible to instrument multimethods with the existing instrumentation features in malli?#2022-02-0818:00Ben SlessWon't multi schema with the dispatch fn itself cover it?#2022-02-0818:24CaseyPerhaps yes.. but then I have to define all schemas on multi-schema on the dispatch fn itself. I'd rather keep the schemas with the defmethod rather than defmulti.#2022-02-0818:58ikitommiHow would you like to define the schemas with defmethod?#2022-02-0822:04CaseyThat's a good question.. defmethod doesn't support metadata afair..only the defmulti does. :thinking_face: Hmm#2022-02-0821:55crimeministerI've got a user-supplied value stored in a map that is a malli schema. I would like to validate that the schema itself is valid; is there a predefined way to do so, or should I author my own "meta-schema" for checking my schema data?#2022-02-0917:31bringeHello. I’m wondering how I can update a schema to dissoc a prop if the schema uses :and at the top-level. For example, if I have the following:
(def my-schema
    [:and
     [:map
      [:id uuid?]
      [:some-prop int?]
      [:some-prop-2 int?]]
     [:fn {:error/message "Some prop must be less than some prop 2"}
      (fn [{:keys [some-prop some-prop-2]}]
        (< some-prop some-prop-2))]])
and I want to dissoc :id from the map, but keep everything else in the schema the same, how can I do that? I see the malli.util dissoc function and others, but I’m not sure how to use them when :and is involved.
#2022-02-0917:46bringeAh, it seems that (mu/update my-schema 0 mu/dissoc :id) works, but maybe there is a better way?#2022-02-0918:08crimeministerThere's a function for manipulating properties that might be worth looking at in case it makes intent clearer: (mu/update-properties).#2022-02-0919:23pithylessfor mu/merge there is a kind of special case: > * for :and schemas, the first child is used in merge, rest kept as-is I wonder if that could (and should) be generalized to other malli.util operations#2022-02-1003:22bringe@U066SF64X I’m not sure if update-properties can be used directly in this case, given the :and , but maybe there’s a good way to use it in combination with another operation to make intent clearer. @U05476190 Yeah :thinking_face:.#2022-02-0920:08oliver marksHI, I have this spec which is being used by a reitit route, I am having issues where it fails because the value is a string instead of a keyword, I have been trying to use the json transformer but I think it fails because its not specifically a keyword test, any suggestion on how I could solve this ?
[:or
   [:map {:closed true}]
   [:map {:closed true}
    [:start [:enum :january :feburary :march :april :may :june :july :august :september :october :november :december]]
    [:end [:enum :january :feburary :march :april :may :june :july :august :september :october :november :december]]]]
#2022-02-0920:22Ben SlessYou need to add a decoder#2022-02-0920:28oliver marksSo i am running the spec through the json-transformer decoder but I dont think it handles this case because its of enum type not keyword#2022-02-0920:29oliver marksI am currently looking at -json-decoders trying to see if I can add to that some how#2022-02-0920:32oliver marks
[:category/type {:optional true} keyword?]
currently the above is working as intended, feels like i need to extend some where I tried adding [:and keyword? [:enum :one :two]] but that does not trigger the conversion
#2022-02-0921:19Grant HornerIs there a :seqable schema that would allow you to specify the children’s type? I’m running into an issue where I’d like to have a function accept a vector, list or set of a particular entity, but haven’t been able to figure out how to build the correct collection schema for it. I’m assuming I should be using malli.core/-collection-schema to add a custom schema to the registry#2022-02-1108:12ikitommiYou can also just define the Schema as Var as use it without registering, like Reagent compoents:
(def Seqable (m/-collection-schema {:type 'Seqable, :pred seqable?}))

(m/validate [Seqable :int] '(1 2 3)) ; => true
#2022-02-1108:12ikitommiIf you feel that is important, please write an issue to add that to malli.#2022-02-1108:14ikitommiadding new types means the following: • add it to the core • generator for it • JSON Schema (and optionally swagger) mappings • default humanized error message • clj-kondo mappings • + tests#2022-02-1108:16ikitommiwithout registering it, you won’t can’t deserialize a serialized schema, but for anything else, it should work normally.#2022-02-1005:42ZaymonHello. Is there a way to make the exception-info created by Malli coercion less verbose? Currently 500 lines of output to the terminal is a bit hard to parse, mostly because the exception info includes a lot of information about the request. Either that or a better way to print only the relevant parts of the exception would also be appreciated.#2022-02-1005:46Zaymon#2022-02-1005:58Zaymon
(def malli-coercer
  (malli-coercion/create
   {;; set of keys to include in error messages
    :error-keys #{:humanized
              ;;     :type
              ;;     :coercion
              ;;     :in
                  :schema
                  :value
                  :errors
                  #_:transformed}
   ;; schema identity function (default: close all map schemas)
    :compile mu/closed-schema
   ;; validate request & response
    :validate true
   ;; top-level short-circuit to disable request & response coercion
    :enabled true
   ;; malli options
    :options nil
    :muuntaja formats/instance}))
My Coercion is setup like this
#2022-02-1019:33emccuemalli/humanize?#2022-02-1101:43ZaymonNot sure how to apply that to exception-info#2022-02-1712:07jussiPondered the same the other day, ended up with simplistic helpers
(defn- path->str
  [path]
  (string/join " -> " (map #(if (number? %)
                              (str "[" % "]")
                              (name %))
                           path)))

(defn pretty-schema-error
  "Make schema errors a bit more readable. Handles nil's and formats path."
  [error]
  (let [path (:path error)
        type (:type error)]
    {:path (path->str path)
     :type (if (nil? type) "nil" (name type))})) 
Using it in our reitit API when handling :reitit.coercion/request-coercion errors for example.
#2022-02-1108:17ikitommiwould this make sense? https://clojureverse.org/t/using-schema-like-schemas-in-malli/8613/2?u=ikitommi#2022-02-1108:37jeroenvandijkLooks useful! - a small step for people acquinted with Plumatic Schema to try Malli - offers potentially a soft migration path from Schema to Malli. Assuming enough/all Schema predicates will be covered - extend existing Schema code bases with the power of Malli (I believe :multi will have better error reporting than schema.core/either for instance). This again assumes that the translation of Schema will be near to perfect#2022-02-1108:18ikitommi
(require '[malli.experimental.lite :as l])

(def Schema
  (l/schema
   {:map1 {:x int?
           :y [:maybe string?]
           :z (l/maybe keyword?)}
    :map2 {:min-max [:int {:min 0 :max 10}]
           :tuples (l/vector (l/tuple int? string?))
           :optional (l/optional (l/maybe :boolean))
           :set-of-maps (l/set {:e int?
                                :f string?})
           :map-of-int (l/map-of int? {:s string?})}}))
;[:map
; [:map1
;  [:map
;   [:x int?]
;   [:y [:maybe string?]]
;   [:z [:maybe keyword?]]]]
; [:map2
;  [:map
;   [:min-max [:int {:min 0, :max 10}]]
;   [:tuples [:vector [:tuple int? string?]]]
;   [:optional {:optional true} [:maybe :boolean]]
#2022-02-1108:19ikitommi18 lines of optional sugar, for simple cases like defining route parameters with reitit.#2022-02-1108:46jeroenvandijkI can imagine this dsl would not cover complex cases of Malli, but sure looks useful to remove some boilerplate. Maybe it becomes confusing if these way of schema writing get mixed, not sure
#2022-02-1108:55ikitommiyes, I would. not use this for anything “normal”, but for specific / inline cases, good: default reitit+malli:
:parameters {:query [:map 
                     [:x int?]
                     [:y {:optional true} string?]]}
adding reitit+malli support for lite too:
:parameters {:query {:x int?, :y (l/optional string?)}}
#2022-02-1109:15jeroenvandijkI wanted to say maybe instead of supporting lite directly, asking the user to use to dsl themselves might lead to less confusion. But I see malli can go into malli-lite and malli-lite can go into malli :)
:parameters {:query {:x int?, :y (l/optional [:map-of string? string?]]}
Generates a valid malli schema, and also the other way around:
:parameters {:query [:map 
                     [:x int?]
                     [:y {:optional true} (map-of string? string?]]}
Nice 💪
#2022-02-1109:18jeroenvandijkSo that actually means complex cases are also covered, because normal malli can be used where necessary?#2022-02-1109:59ikitommiyes.#2022-02-1114:15ikitommimalli.experimental.lite now merged in master.#2022-02-1109:59Ben Slesshttps://www.asyncapi.com/ 👀#2022-02-1317:42ikitomminot sure how this relates to malli, but 👍#2022-02-1317:52Ben Slessmostly, thinking about a good way to bring this together with malli, hopefully finding a lossless translation between the two. AsyncAPI looks like a very interesting way to solve the "schema problem" at an organizational level + all the metadata I would have had to reinvent the wheel to specify, which could go beyond interfacing with malli, but further code generation. You could derive reitit routes from it, and more#2022-02-1114:15ikitommimalli.experimental.lite now merged in master.#2022-02-1115:54Brett RowberryToday, when humanizing a value against a map schema, the error message "invalid type" isn't super useful. I know I can add a custom error message. How could it be more descriptive?
(malli.error/humanize (malli.core/explain [:map] "string"))
;; => ["invalid type"]
#2022-02-1116:18dvingohttps://github.com/metosin/malli#custom-error-messages#2022-02-1122:22Brett RowberryRight, I know I can do that for a given schema, but wondered if maybe all maps could say something more specific.#2022-02-1211:41ikitommiYou can override the default error for invalid type, using options to me/humanize.#2022-02-1211:42ikitommithere should be an example behind the link..#2022-02-1212:34dvingooh woops sorry @U021RHDFFHN overlooked that you said that. I learned something, just tried this out and it works:
(malli.error/humanize
    (malli.core/explain [:map] "string")
    {:errors (-> default-errors
               (assoc ::m/invalid-type
                      {:error/fn (fn [{:keys [value schema] :as in} _]
                                   (str "The value you provided: '" value "' is not the correct type for the schema: '" (m/form schema) "'"))}))})
=> ["The value you provided: \"string\" is not the correct type for the schema: ':map'"]
#2022-02-1317:41ikitommithe (m/form schema) could be (m/type schema), just report on type, not whole form. otherwise, 💯#2022-02-1418:16Brett RowberryThanks! This works best for my use case:
(malli.error/humanize (malli.core/explain [:map] nil))
;; => ["invalid type"]

(malli.error/humanize
 (malli.core/explain [:map [:hi string?]] nil)
 {:errors (-> malli.error/default-errors
              (assoc ::m/invalid-type
                     {:error/fn (fn [{:keys [_value schema]} _]
                                  (str "The value provided does not conform to schema: '" (m/form #_m/type schema) "'"))}))})
;; => ["The value provided does not conform to schema: '[:map [:hi string?]]'"]
#2022-02-1511:00Martynas MHow does one install sci as an external dependency? I try to run tests of malli and they fail with this error: :malli.core/sci-not-available Should I include it into my ~/.clojure/deps.edn?#2022-02-1511:00Martynas MThis was the info from #babashka channel. But it's not enough context.#2022-02-1511:01borkdudehttps://github.com/metosin/malli#running-tests#2022-02-1511:01borkdudeIt seems you need the :test alias#2022-02-1511:02borkdudeI recommend upgrading SCI to 0.3.0 for better performance#2022-02-1511:03Martynas MHow would one run tests from emacs(spacemacs+CIDER)? If I understand correctly you'd need to run it a JS-Clojure REPL. It failed for me when I used Java-Clojure REPL.#2022-02-1511:07borkdude@invertisment_clojuria You can place a file .dir-locals in the repo:
((clojure-mode
  (cider-clojure-cli-aliases . "test")))
#2022-02-1511:07borkdudeand then cider-jack-in , after that you should be able to run tests from CIDER#2022-02-1511:08borkdudeif you do not want to add the file, you can do it like this:
C-u M-x cider-jack-in
#2022-02-1511:08borkdudeand then choose clojure-cli and then add the test alias here: -M:test:cider#2022-02-1511:08borkdudeand press enter#2022-02-1511:08Martynas Mok.#2022-02-1511:11Martynas MWhat should I choose when it says "connect sibling cljs"? There are many options. node, figwheel, shadow and others#2022-02-1511:12borkdudeDunno, I don't ever use this. If I want to run tests, I do it directly from the Node REPL#2022-02-1511:12Martynas MSo you develop without a REPL?#2022-02-1511:13Martynas MI'm used to have everything under one process in Java. That's why I ask. I know that in CLJS it's different.#2022-02-1511:14Martynas MSo should it be ./bin/node and that's it? And then you'd proceed to go and change the sources?#2022-02-1511:14Martynas MAnd have the REPL running just in case?#2022-02-1511:15borkdudeI do use the REPL, but for CLJS I found it too brittle to do it from my editor so I chose to simplify things. I use (require '[ns] :reload) in the REPL when I want to reload changes, and invoke test expressions or files when I want to test something. This is a spartan workflow, but it's the least time consuming for me personally.#2022-02-1511:17Martynas MShould I also add how to start-up the emacs REPL into the README? I'll make a PR for a bugfix.#2022-02-1511:18borkdudeI was just providing info, I'm not the maintainer of this project, so I'll let @ikitommi or whoever decide on that :)#2022-02-1513:27dvingoI've had success running the cljs tests in malli repo in a repl, but I augmented the shadow-cljs.edn file to load one of the app targets in a browser, and then when the tests were working in the browser I verified them for node by running ./bin/node this was from cursive though. it should also work with cider with the relevant shadow integration#2022-02-1710:34DrLjótssonIs there a way to specify that a set needs to have at least one element? [:+ ...] only seems to work for sequences#2022-02-1710:43DrLjótssonThis is what I have come up with
[:and
 [:set :int]
 [:fn (fn [x] (not (empty? x)))]]
#2022-02-1711:11ikitommimaybe [:set {:min 1} :int]?#2022-02-1711:37DrLjótssonThat’s it, works!#2022-02-1711:37DrLjótssonThanks#2022-02-2019:48oliver marksHi, hoping someone can point me in the right direction, can you extend json-transformer-provider https://github.com/metosin/reitit/blob/198cfda00d20093f3d7b3069e5e902835c396698/modules/reitit-malli/src/reitit/coercion/malli.cljc#L35 I basically want to convert a specific keys value to a keyword the malli spec is an enum of keywords and the json decoder is not converting the string to a keyword for me so I need some custom logic here I believe, but I am unsure how to inject this ?#2022-02-2214:48rovanionI'm getting data to a Reitit handler with malli coercion from a simple HTML form. One of the fields should either be a number or be left blank when the form is submitted. I want to translate that to either nil or to remove the key entirely from the map during coercion, is that possible?#2022-02-2214:50rovanionRight now coersion fails because the feild holds a string when it should hold an int, but just defining the value of the field to be [:or string? int?] means that all other functions have to deal with the possibility of that field being a non-int.#2022-02-2215:21dvingoyou could make your own schema type and then add custom coercion logic there. For example this is a custom schema for local dates:
:local-date (m/-simple-schema
                            {:type            :local-date
                             :pred            tu/date?
                             :type-properties {:gen/gen gen-date
                                               :decode/string (fn [v]
                                                                (log/info "Decoding to local-date: " v)
                                                                (tu/date v))}})
and add that to your malli registry
#2022-02-2215:23dvingoand to deal with the logic of conditionally removing the key from a map, my guess is that you could change the malli coercer: https://github.com/metosin/reitit/blob/master/doc/coercion/malli_coercion.md#configuring-coercion by changing the transformers#2022-02-2215:43rovanionInteresting, I like that first idea. Make a type with a name like :nilable-form-int, simple enough. Thank you! Writing a coercer would probably be better, but I don't think I could wrap my head around it fast enough to still please the deadline.#2022-02-2215:55ikitommi@rovanion maybe:
(def NilableInt
  [:maybe {:decode/string (fn [x] (when-not (str/blank? x) (mt/-string->long x)))} :int])

(m/decode
 [:map
  [:x NilableInt]
  [:y NilableInt]]
 {:x ""
  :y "123"}
 (mt/string-transformer))
; => {:x nil, :y 123}

(mg/sample
 [:map
  [:x NilableInt]
  [:y NilableInt]])
;({:x -1, :y nil}
; {:x nil, :y nil}
; {:x 0, :y -1}
; {:x nil, :y nil}
; {:x 2, :y nil}
; {:x -3, :y -1}
; {:x -2, :y nil}
; {:x 3, :y 1}
; {:x nil, :y nil}
; {:x -8, :y -1})
#2022-02-2216:02rovanionHah, was just writing that exact :decode/string-function in my own simple-schema. But yeah, that fits the bill!#2022-02-2216:03rovanionI really like that Clojure has functions like when-not in the standard library so I don't have to make them up in some util library.#2022-02-2216:07rovanions/functions/functions and macros/#2022-02-2217:31rovanionThis is where I'm at now:
(register! :html/nilable-int                                                                                          
  (malli/-simple-schema                                                                                               
   (fn [opts _]                                                                                                       
     {:type            :html/nilable-int                                                                              
      :pred            (some-fn int? nil?)                                                                            
      :property-pred   (malli/-min-max-pred nil)                                                                      
      :description     "An int or nil. Empty string is transformed to nil."                                           
      :type-properties {:decode/string (fn [x] (when-not (string/blank? x)                                            
                                                 (mtransform/-string->long x)))                                       
                        :gen/gen (gen/large-integer* opts)}})))
I just have to write a generator that generates nil sometimes. Sadly :gen/schema [:maybe [int? opts]] doesn't respect :min passed in opts.
#2022-02-2218:12ikitommiint? is just a predicate, use :int , which is a real schema#2022-02-2218:15rovanionThe practical implications of the distinction is lost on me, I thought both were resolved to the exact same schema. Either way it made no difference, setting :gen/schema [:maybe [:int opts]] still generated negative numbers when opts was {:min 0}.#2022-02-2218:59ikitommi
(mg/sample [:any {:gen/schema [:int {:min 100}]}])
; => (101 101 101 101 102 108 105 100 111 148)
#2022-02-2219:01ikitommi
(mg/sample [:maybe [:any {:gen/schema [:int {:min 100}]}]])
; => (nil nil nil 102 nil nil 102 103 111 159)
#2022-02-2219:19rovanionSorry, I must have failed to press C-c C-c or something after switching from int? to :int because it's working now and I don't know what else could have caused it to fail. Thank you so much for your time.#2022-02-2219:20rovanionThis is the schema I ended up with, does both generate, transform and conform as I want it:
(register! :html/nilable-int                                                                                  
  (malli/-simple-schema                                                                                       
   (fn [opts _]                                                                                               
     {:type            :html/nilable-int                                                                      
      :pred            (some-fn int? nil?)                                                                    
      :property-pred   (malli/-min-max-pred nil)                                                              
      :description     "An int or nil. Empty string is transformed to nil."                                   
      :type-properties {:decode/string (fn [x] (when-not (string/blank? x)                                    
                                                 (mtransform/-string->long x)))                               
                        :gen/schema [:maybe [:int opts]]}})))
#2022-02-2220:17oliver marksSo been playing a bit more and built up some snippet code to show my issues, the first works as I expect the second part does not seem to transform in the same way and I can not figure out why, can any one lend some assistance ?
(def my-spec
  [:map {:closed true}
   [:object/type [:keyword {:json-schema/type "keyword" :string-schema/type "keyword"}]]])

(m/decode
 my-spec
 {:object/type "test"}
 mt/json-transformer)

;; => #:object{:type :test}
How ever using this returns an error complaining that objecct/type is not a keyword, I was under the imrpression it should be converted the same as m/decode, so I am obviously missing a trick some where, tried a few things but just can not get it to decode.
(reitit.coercion.malli/create
 {:transformers {:body {:default reitit.coercion.malli/default-transformer-provider
                        :formats {"application/json" reitit.coercion.malli/json-transformer-provider
                                  "application/transit+json; charset=utf-8" reitit.coercion.malli/json-transformer-provider
                                  "application/transit+json" reitit.coercion.malli/json-transformer-provider}}
                 :keyword {:default reitit.coercion.malli/string-transformer-provider}
                 :string {:default reitit.coercion.malli/string-transformer-provider}
                 :response {:default reitit.coercion.malli/default-transformer-provider}}})
#2022-02-2309:25olyjust some thoughts in case its related, I am using transit+json because my keys are namespaces this is so they get preserved over the wire#2022-02-2314:05lambderhello#2022-02-2314:05lambderI tried to apply this https://github.com/metosin/malli/pull/545/files#2022-02-2314:05lambderin my project#2022-02-2314:06lambderI've added zoned-date-time#2022-02-2314:06lambderin some other ns I define schema like:
(def ZDT
  (mt/time-schema :zoned-date-time))
#2022-02-2314:07lambderthen in the reitit router I have malli coercion set and the endpoint has this:#2022-02-2314:08lambder
:parameters
  {:body
   [:map
    [:scrape-request
     [:map
      [:id int?]
      [:from-date
       [:map
        {:registry
         #:my-name-space{:zdt :zoned-date-time}}
        :my-name-space/zdt]]]]]},
#2022-02-2314:08lambderbut when I try to create the routes I get:#2022-02-2314:09lambder
Execution error (ExceptionInfo) at reitit.exception/exception (exception.cljc:19).
:malli.core/invalid-schema {:schema :zoned-date-time}

{:schema :zoned-date-time}
#2022-02-2314:09lambderany idea?#2022-02-2314:10lambdersame problem if I don't use registry#2022-02-2314:10lambder
:parameters
  {:body
   [:map
    [:scrape-request
     [:map [:id int?] [:from-date :zoned-date-time]]]]},
#2022-02-2314:11lambderwhat am I doing wrong?#2022-02-2314:11lambderplease?#2022-02-2314:45lambderI probably son't use the default registry correctly. Any hints please?#2022-02-2315:03pithylessInstead of :zoned-date-time try referring directly to your var ZDT#2022-02-2315:04pithylessIf you're using the default registry, you need to make sure it registers :zoned-date-time as a known schema. See: https://github.com/metosin/malli/blob/2398df55ee806e25592fabf4d0c642ee3a2b233f/src/malli/core.cljc#L2371-L2378#2022-02-2315:21lambder@U05476190 thanks, I did small step forward. The route construction no longer fails after I defined the custom registry via :coercion
(defn service-routes []
  ["/api"
   {:coercion   (reitit.coercion.malli/create
                  (merge reitit.coercion.malli/default-options
                    {:options {:registry (merge
                                           (m/default-schemas)
                                           (malli-time/time-schemas))}}))
#2022-02-2315:21lambderbut , on Swagger UI I'm getting:#2022-02-2315:22lambder\#2022-02-2315:24lambderthe swagger.json is:
#2022-02-2315:25lambderI 'believe it should be:
"type": "string",
"format": "date-time" 
no?
#2022-02-2315:27pithylessSorry, no idea; probably best to ask outside of this thread - I'm not familiar with swagger/malli integration#2022-02-2315:36lambderwill try to update https://github.com/metosin/malli/pull/545/files with#2022-02-2315:37lambdermalli offers json swagger schema integration#2022-02-2317:05lambderhere @ronny#2022-02-2316:39Ronny LøvtangenThanx @lambder, I was trying to add support for java.time.LocalDate and java.time.Instant to my project, and your post helped me. I added the code from https://github.com/metosin/malli/pull/545/files and imported it to the registry with
(defn service-routes []
  ["/api"
   {:coercion   (reitit.coercion.malli/create
                  {:options {:registry (merge
                                           (m/default-schemas)
                                           (malli-time/time-schemas))}})
(I believe the merge with default-options is not necessary, as reitit.coercion.malli/create already does that) I was then able to coerce a string to java.time.LocalDate by specifying the property as:
[:mydate {:description "My description"
          :json-schema/example "2021-11-05"}
 :local-date]
One thing I noticed, was that if I tried to send a non-parsable value, I got Error: Wrong number of args (2) passed to: slakt-service.malli.time/-&gt;error-reporter/-report--28426 By changing
(defn ->error-reporter
  [parser message]
  (fn -report [value]
to
(defn ->error-reporter
  [parser message]
  (fn -report [value _]
I instead got the message
"Should be local-date or LocalDate"
#2022-02-2317:04lambdergreat stuff, @ronny#2022-02-2317:05lambderhave you seen the massages in the thread?#2022-02-2317:05lambderone thing I'm missing is to make it work with edn request#2022-02-2317:05lambderit works with json only#2022-02-2317:06lambderI'll do
`(fn -report [value & _]
`
#2022-02-2317:06Ronny LøvtangenYes. I was also wondering about type vs format. Not an expert on json schema, but looks like you are right#2022-02-2317:06lambderjust in case some other call needs single arity#2022-02-2317:10Ronny LøvtangenI also added :json-schema/example:
:local-date {:class LocalDate :parser -string->local-date :json-schema/type :local-date :json-schema/example "2022-02-23"}

...

:json-schema/example (:json-schema/example props -name)
Then I don’t have to (but could override if I want to) provide example in my schema.
#2022-02-2317:10lambderif I don't supply the example for date-time, the SwaggerUI will get one form me#2022-02-2317:17Ronny LøvtangenIf I don’t provide json-schema/example, Swagger will show “Unknown Type: local-date”. Maybe your change to
"type": "string",
"format": "date-time" 
is what makes Swagger able to provide an example
#2022-02-2317:17lambderlocal-date is not corret#2022-02-2317:18lambderthere is only date and date-time#2022-02-2317:18lambdersee here https://www.baeldung.com/openapi-dates#2022-02-2317:18lambderfor local dates you probably want to adapt:#2022-02-2317:18lambder
customDate: 
  type: string 
  pattern: '^\d{4}(0[1-9]|1[012])(0[1-9]|[12][0-9]|3[01])$'
  description: Custom date 
  example: "20210130"
#2022-02-2317:23Ronny LøvtangenIsn’t “date” a good fit for java.time.LocalDate? Format yyyy-MM-dd#2022-02-2317:23lambderit is#2022-02-2317:27Ronny LøvtangenWith
:local-date {:class LocalDate :parser -string->local-date :json-schema/type :string :json-schema/format "date"}
Swagger provides an example :thumbsup:
#2022-02-2317:28lambderright#2022-02-2317:28lambderdo you have your app working with edn encoding?#2022-02-2317:29lambderdo I need to provide analog of:#2022-02-2317:29lambder
{:transformers {:body {:default default-transformer-provider
                         :formats {"application/json" json-transformer-provider}}
                  :string {:default string-transformer-provider}
                  :response {:default default-transformer-provider
                             :formats {"application/json" json-transformer-provider}}}
#2022-02-2317:29lambderfor edn?#2022-02-2317:33Ronny LøvtangenNo, sorry, haven't done anything to provide support for edn in our application#2022-02-2318:36lambderwhen my edn request is:#2022-02-2318:36lambder
{
  :scrape-request {
    :id 1
    :from-date #time/zoned-date-time "2022-01-01T16:14:34.447Z"
  }
}
#2022-02-2318:37lambderI'm getting:#2022-02-2318:37lambder
{:schema "[:map {:closed true} [:scrape-request [:map {:closed true} [:id int?] [:from-date :zoned-date-time]]]]", :errors ({:path [:scrape-request :from-date], :in [:scrape-request :from-date], :schema ":zoned-date-time", :value (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z"), :message "Should be zoned-date-time or ZonedDateTime"}), :value {:scrape-request {:id 1, :from-date (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z")}}, :type :reitit.coercion/request-coercion, :coercion :malli, :in [:request :body-params], :humanized {:scrape-request {:from-date ["Should be zoned-date-time or ZonedDateTime"]}}}
#2022-02-2318:37lambdersee: :from-date (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z")#2022-02-2318:37lambderhow on earth this form gets there are all?#2022-02-2318:38lambderthis is added to the registry:#2022-02-2318:38lambder
:zoned-date-time #IntoSchema{:type :zoned-date-time},
#2022-02-2318:39lambderusing this:#2022-02-2318:39lambder
(m/-simple-schema
        {:type type
         :type-properties
               (cond-> {:error/fn         error-fn
                        :decode/json      {:enter safe-parser}
                        :encode/json      {:enter #(prn %)}
                        :json-schema/type (:json-schema/type props -name)}
                 (:json-schema/format props -name) (assoc :json-schema/format (:json-schema/format props -name)))
         :pred pred})
#2022-02-2318:40lambder
the `safe-parser` is:
#2022-02-2318:40lambderprovided by :#2022-02-2318:40lambder
(defn -string->zoned-date-time [x]
  (ZonedDateTime/parse x))
#2022-02-2318:40lambderso this :from-date (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z") looks a lot like (ZonedDateTime/parse x)#2022-02-2318:41lambderbut instead if being invoked is it passed a a value. How did it happen ?#2022-02-2319:15lambderfunny thing the value to decoder is coercion.malli is already wrapped with something like this:#2022-02-2319:16lambder#2022-02-2319:17lambderthis happens in reitit.coercion by transforming the request to the value#2022-02-2319:17lambder#2022-02-2319:36juhoteperi@lambder those decode/json transformations aren't used by coercion here (no coercion really needed for these values) when you are using EDN. As EDN can already represent types, the coercion doesn't do anything. If you have *data-reader* registered for the #time/zoned-date-time tag (or provided :readers option to muuntaja edn format) the value should be correct and the problem must be with the schema validation (the predicate).#2022-02-2319:38lambderhey @juhoteperi no, I don't have the readers supplied to muuntaja sepcially#2022-02-2319:38lambderI got `
:muuntaja   formats/instance
#2022-02-2319:39lambderthat instance is:#2022-02-2319:39lambder
(def instance
  (m/create
    (-> m/default-options
      (update-in
        [:formats "application/transit+json" :decoder-opts]
        (partial merge time/time-deserialization-handlers))
      (update-in
        [:formats "application/transit+json" :encoder-opts]
        (partial merge time/time-serialization-handlers)))))
#2022-02-2319:39juhoteperiMuuntaja will use *data-readers*, which is already provided by Clojure if you have the readers in a data_readers.clj file somewhere#2022-02-2319:40lambderI do in my core ns `
(time-literals.read-write/print-time-literals-clj!)
#2022-02-2319:40juhoteperiBut hmm, it could be problem with those data readers, because :value (. java.time.ZonedDateTime parse "2022-01-01T16:14:34.447Z") seems really strange. Looks like the value isn't the ZonedDateTime instance, but the Clojure form calling that ZonedDateTime/parse method#2022-02-2319:40lambderbut not setting it for read#2022-02-2319:40lambderwill get back to it in about 1h30m#2022-02-2319:41lambderneed to run now#2022-02-2319:41juhoteperi> The library includes the magic file data_readers.cljc which Clojure and the Clojurescript compilers() will look for. #2022-02-2319:41lambderthanks for everything#2022-02-2319:41juhoteperiSo just including time-literals in the classpath should add data-readers for those tags#2022-02-2319:44juhoteperiIf data-readers are working correctly, if you eval in the REPL {:foo #time/zoned-date-time "2022-01-01T16:14:34.447Z"} you should get the same map back, the tagged value is read to ZonedDateTime instance, and because you have the print method registered, it should be printed out the same way.#2022-02-2319:55lambderIn need to add time readers to end/read-string option for it to work#2022-02-2319:56lambderSo I guess they are not added to data_readers, but I'll dbl check#2022-02-2323:02lambder@juhoteperi this is just black magic to me#2022-02-2323:02lambder#2022-02-2406:41henryw374@lambder above issue fixed on the latest release - see https://github.com/henryw374/time-literals readme for details. tl;dr it was basically a workaround for a problem with Clojurescript that has taken some time to get fixed - see https://clojure.atlassian.net/browse/CLJS-3294#2022-02-2408:47lambder@henryw374 Thank you !#2022-02-2409:40lambderthis worked like a charm 😄#2022-02-2411:01rovanion@ikitommi I wrote a version of -min-max-pred that can deal with nil being passed to it as I could not figure out an f that could transform nil into a number that would make sense min and max to apply to. If you think you have use for it anywhere in Malli, here it is:
(defn -nilable-min-max-pred [f nilable]
  (fn [{:keys [min max]}]
    (cond
      (not (or min max))    nil
      (and  min max f nilable) (fn [x] (if (nil? x) true (let [size (f x)] (<= min size max))))
      (and  min max f)         (fn [x] (let [size (f x)] (<= min size max)))
      (and min max nilable)    (fn [x] (if (nil? x) true (<= min x max)))
      (and min max)            (fn [x] (<= min x max))
      (and min f nilable)      (fn [x] (if (nil? x) true (<= min (f x))))
      (and min f)              (fn [x] (<= min (f x)))
      (and min nilable)        (fn [x] (if (nil? x) true (<= min x)))
      min                      (fn [x] (<= min x))
      (and max f nilable)      (fn [x] (if (nil? x) true (<= (f x) max)))
      (and max f)              (fn [x] (<= (f x) max))
      (and max nilable)        (fn [x] (if (nil? x) true (<= x max)))
      max                      (fn [x] (<= x max)))))
And its use in my code is just:
(register! :html/nilable-int
  (malli/-simple-schema
   (fn [opts _]
     {:type            :html/nilable-int
      :pred            (some-fn int? nil?)
      :property-pred   (su/-nilable-min-max-pred nil true)
      :description     "An int or nil. Empty string is transformed to nil."
      :type-properties {:decode/string (fn [x] (when-not (string/blank? x)
                                                 (mtransform/-string->long x)))
                        :gen/schema [:maybe [:int opts]]}})))
#2022-02-2513:34oliver marksso still struggling with transformers not being applied, I am looking at the request chain I can see that :body-params is populated with a namespaced map, how ever in the next steps I get this
+:body-params {:object/type "test"}}

--- :request---

  nil

--- :request :reitit.ring.coercion/coerce-exceptions ---

  {}

--- :request :reitit.ring.coercion/coerce-response ---

  {}

--- :response---

  {:body {:coercion :malli,
          :errors ({:in [],
                    :message "invalid type",
                    :path [],
Seems a bit strange that the :in would be blank, I am getting this from a test which I am using to help me figure out my issue also invalid type is not over descriptive in this context seems, any tricks to getting more info from malli on what its doing, or does this help any one give me some suggestions ?
#2022-02-2513:50oliver marksI would also be intrested in a bit more info on this map, https://github.com/metosin/reitit/blob/198cfda00d20093f3d7b3069e5e902835c396698/modules/reitit-malli/src/reitit/coercion/malli.cljc#L110 are there any docs on what going on is :transformers the function that gets applied to produce the body-params ie is it this step which should be converting my strings to keywords based on the schema ?#2022-02-2516:34Grant HornerRunning into a weird issue: I’m getting the following report on a function call in my test suite:
{:in [1 0 :foo-id],
   :message "should be a positive int",
   :path [1 0 :foo-id],
   :schema pos-int?,
   :value 9218}
where, according to the report, the value is in fact a positive integer. The function is expecting a
[:sequential [:map [:foo-id pos-int?] [:name string?]]]
as an input, and is receiving
({:name "Foo Inc", :foo-id 9218, :type "security-id", :value 9218})
as the arg. This correctly passes the call to malli.core/validate , but is failing in my test suite for some reason
#2022-02-2516:47Grant Hornerfalse alarm. the value is actually a biginteger. maybe the error message could be improved in these cases? perhaps including the value’s type?#2022-02-2519:35oliver marksI may have had a bit of progress, I seem to be able to make it work using application/json in place of application/transit+json I was using the later as cljs-ajax drops the namespace in the keys which I was hoping to avoid. could this be a case of the flow is different for application/transit+json I did add it to the coercsion map like below.
{:transformers {:body {:default reitit.coercion.malli/default-transformer-provider
                          :formats {"application/json" reitit.coercion.malli/json-transformer-provider
                                    "application/transit+json" reitit.coercion.malli/json-transformer-provider}}
#2022-02-2600:23sound2gdhi clojurians~ I'm stucking on decoding json from string with malli#2022-02-2600:23sound2gd
(require '[malli.core :as m])
  (require '[malli.transform :as mt])

  (m/decode [:map
             [:x :string]
             [:y :int]]
            "{\"x\": \"aa\",\"y\": 3}"
            (mt/transformer
             mt/json-transformer))
;; => "{\"x\": \"aa\",\"y\": 3}"
#2022-02-2606:16Ben SlessJson transformers assume the data is already deserialized#2022-02-2606:17Ben SlessYou need a library like jsonista for that#2022-02-2614:35sound2gdthanks. finally work out like this#2022-02-2614:35sound2gd
(m/decode [:string {:decode/string #(json/decode % true)
                      :encode/string json/encode}]
            "{\"x\": \"aa\",\"y\": 3}"
            (mt/transformer
             mt/strip-extra-keys-transformer
             mt/default-value-transformer
             mt/string-transformer))
#2022-02-2815:04rovanionGiven a Malli map schema like [:map [:key {:optional true} :int]], how do I get at the properties assigned the keyword :key in the map when writing a walker? In this example, these are {:optional true}. As you can see in the below example the walker visits all the "values" of the map, in this case just the :int schema, and then out to the :map schema. Is there any way to get to the inbetween step, the "schema" and properties of :key?
(malli/walk                                                                                                   
      [:map [:key {:optional true} :int]]                                                                          
      (fn [schema path children options]                                                                           
        (println "Schema:" schema)                                                                                 
        (println "Path:" path)                                                                                     
        (println "Children:" children)                                                                             
        (println "Options:" options)                                                                               
        (println "Properties:" (malli/properties schema))                                                          
        (println))) => nil 


    ;; Schema: :int                                                                                                 
    ;; Path: [:key]                                                                                                  
    ;; Children: nil                                                                                                 
    ;; Options: nil                                                                                                  
    ;; Properties: nil                                                                                               
    ;;                                                                                                               
    ;; Schema: [:map [:key {:optional true} :int]]                                                                   
    ;; Path: []                                                                                                      
    ;; Children: [[:key {:optional true} nil]]                                                                       
    ;; Options: nil                                                                                                  
    ;; Properties: nil   
#2022-02-2815:29ikitommiWould this help? https://cljdoc.org/d/metosin/malli/0.8.3/doc/tips#walking-schema-and-entry-properties#2022-02-2816:22rovanionAh, okey, so you need to pass {::malli/walk-entry-vals true}.#2022-02-2816:22rovanionThanks.#2022-02-2820:00oliver marksSo setup a test case from the example project, https://github.com/olymk2/reitit/commit/6fd0ee5b6d1553cfc25227836b8a3a126b77b5d5 I have added a single param to the malli spec if you just submit the defaults it sends "january" and errors because its not a key my understanding is this should be decoded by the transformer ? is this a bug or something I am not understanding ?#2022-03-0101:01vemvIs there a supported way to introspect that the type of the following schema is string??
[:fn {:description "An ID"}
 string?]
malli.core/type returns :fn for it, which is not as useful. ...`(m/type string?)` does return string? (as a symbol), so I'd say that my expectation is reasonable
#2022-03-0101:36vemvI had some luck with considering this vector to be "just data" but yeah... an official helper would be reassuring#2022-03-0104:32Ben SlessMy unofficial hot take is fn specs are evil for this very reason#2022-03-0117:15CaseyIs it not possible to pass an options map (containing a registry) to malli.util/update-in ? The other functions in that ns take options map, but this one doesn't appear too. update also lacks an options arg. I can't use a local registry because I've a non-map (primitive) registry.#2022-03-0213:19DenisMcHi, I would like to disallow null bytes in :string fields in my malli schema because Postgres can’t handle them. What is the cleanest way to add this sort of validation? Thanks in advance.#2022-03-0213:43DenisMcJust looking at this further: I’m running into this null byte generator issue because I need some of my strings to be non-empty, so I’m setting :min (and :max) parameters in Malli, like so:
[:my-param [:string {:min 1 :max 5}]]
I see that the malli.generator/-string-gen code looks like this:
(defn- -string-gen [schema options]
  (let [{:keys [min max]} (-min-max schema options)]
    (cond
      (and min (= min max)) (gen/fmap str/join (gen/vector gen/char min))
      (and min max) (gen/fmap str/join (gen/vector gen/char min max))
      min (gen/fmap str/join (gen/vector gen/char min (* 2 min)))
      max (gen/fmap str/join (gen/vector gen/char 0 max))
      :else gen/string-alphanumeric)))
..so the generator generates alphanumeric strings if the min or max length are not set, but generates random characters (including non-alphanumeric characters) if the length is set. I’m wondering what the logic is for this? Why does the string length determine what sort of characters are included in the randomly generated strings?
#2022-03-0412:05ingesolHi! I’m trying to get malli 0.8.4 to generate clj-kondo config for my browser CLJS project, trying this code
(ns mynamespace
(:require-macros [malli.dev.cljs :as dev]))

(defn test-fn
  "return y when x is larger than 5"
  {:malli/schema [:=>
                  [:cat :int :keyword]
                  [:maybe :keyword]]}
  [x y]
  (when (< 5 x)
    y))

(dev/start! nil)

(test-fn 1 2)
Instrumenting seems to work and I’m getting the expected validation error in my browser console, but I’m seeing no clj-kondo config being generated for my function schema, only seeing this in .clj-kondo/metosin/malli-types:
{:linters {:unresolved-symbol {:exclude [(malli.core/=>)]}}}
#2022-03-0716:16dvingoit looks like cljs support broke during a refactor of how function schemas are stored (there is now a map for :clj and one for :cljs) so the kondo data is looking up :clj even in :cljs code: https://github.com/metosin/malli/blob/002d5cbc724e83f07f126d66b19df022859f1cbf/src/malli/clj_kondo.cljc#L179 https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2385 I should have some time over the next few days to send a pr to address this#2022-03-0717:25ingesolRight, that’s what I suspected too, since the CLJS support was pretty fresh. Thank you so much for looking into it!#2022-03-0416:02Martynas MHey. Is there a way to generate a value from a predefined list of values? Or maybe it's possible to "remember" previously generated values so that they would be able to be reused to produce random connections (as in DB sense)?#2022-03-0716:46ikitommimaybe:
(mg/sample [:any {:gen/elements [1 2 3]}])
; => (1 1 3 2 1 1 2 1 1 1)
#2022-03-0416:21mafcocincoSimple newbie question: For malli.generator/generate is there a way to force malli to generate all fields, including optional?#2022-03-0716:45ikitommiat the moment, no. But would be 1-2 extra lines of code: • new option :mg/generate-optional-values • read it here https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L90 - if true always, if false never, default to nil (current) • add a test • document int README#2022-03-0518:48match37Using malli 0.8.4, got below error, what did I miss? Appreciate your help!
user=> (require '[malli.core :as m ])
nil

user=> (def s (->> [:union
                    [:map [:x :string]]
                    [:schema [:map [:x :int]]]]))
#'user/s

user=> (m/form s)
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
:malli.core/invalid-schema {:schema :union}

user=> (m/validate s {:x "x"})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
:malli.core/invalid-schema {:schema :union}
#2022-03-0604:51match37answer my own question, I need register some predefined schemas.
{:registry (merge (mu/schemas) (m/default-schemas))}
#2022-03-0714:04henrikIs it possible to spec multi arity functions with the experimental Plumatic-style schema? https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-inline-schemas#2022-03-0714:08ikitommisure:
(mx/defn plus
  ([x :- :int] x)
  ([x :- :int, y :- :int] (+ x y))
  ([x :- :int, y :- :int, & zs :- [:* :int]] (apply + x y zs)))
#2022-03-0714:09henrikDoh, naturally, thank you.#2022-03-0714:09ikitommi👍#2022-03-0813:14henrikMy head was not screwed on right yesterday. What I meant to ask was, how do I scope a particular return spec to a particular arity. In e.g. Snoop, this is possible since the return spec is part of the arity. In Plumatic, I'm unsure what the syntax is.#2022-03-0815:05ikitommithere is no such thing in the Plumatic Syntax. Thought of adding that after the args-vector, but then it would not be Plumatic Syntax anymore.#2022-03-0815:05ikitommiI propose to ask this from the plumatic schema gang via an issue, happy to incorporate whatever they think is good, could add my 2 cents in the conversation there#2022-03-0815:29ikitommi… and you can do that with non-plumatic syntax already.#2022-03-0815:30ikitommi
(defn f1
  "doc"
  (^{:malli/schema [:=> [:cat :int] :int]} [x] (inc x))
  (^{:malli/schema [:=> [:cat :int :int] :int]} [x y] (+ x y)))
or
(defn f2
  "doc"
  {:malli/schema [:function
                  [:=> [:cat :int] :int]
                  [:=> [:cat :int :int] :int]]}
  ([x] (inc x))
  ([x y] (+ x y)))
#2022-03-0908:26henrikI think it would be good to alias the Plumatic macro name to something like >defn as well. Partly because it seems to be a bit of a convention, partly because it makes it possible to refer it without breaking the equivalent in clojure.core.#2022-03-0908:28henrikRegarding arity-specific return specs; to me it makes sense to tack them on to the param vectors. It might even be possible to combine the two: a spec that runs regardless of the arity, and a spec that runs per arity.#2022-03-0909:44ikitommiPersonally, I really don’t like the >defns. clojure.test is the only ns that I require macros without namespace alias. mx/defn tells where it is coming from.#2022-03-0909:45ikitommi
(mx/defn plus :- :int ;; the default, masked by 2 arities? 
  ([x :- :int] :- :int x)
  ([x :- :int, y :- :int] (+ x y))
  ([x :- :int, y :- :int, & zs :- [:* :int]] :- :int (apply + x y zs)))
#2022-03-0909:47ikitommistarting to be quite verbose. Have asked if Cursive could grey out the schema hints so it would be move visbile#2022-03-0911:10henrik> Personally, I really don’t like the >defns Fair enough. I tend to lean towards referring, when it's canonical or conventional (we decide on one >defn, and overloading it is an error), and >defn has become something of a naming convention among function spec libraries. > verbose Agreed, and it's also not that easy to make a formatter understand the syntax to begin with. The separate vector approach of Snoop/Guardrails etc. has the advantage of registering as a conventional function in terms of arguments fed to it. By contrast, Plumatic seems like it requires quite a bit of interpretation of syntax, especially when extended in this way. It also has advantages, of course.#2022-03-0903:53Ryan TateHello, Malli looks very cool. Is it possible in Malli to retain values generated during validation like spec/conform and spec/conformer? Or if not maybe you can think of some Malli-ish way to solve this: I have some expensive validations, for example I accept an xpath as input and validate by compiling it. I don't want to throw the resulting value away and have to re-generate it later. So I transform (with s/conformer) the xpath string to a map with a compiled version (object) of it during validation and this map is the result of validation (with s/conform):
(require '[clj-xpath.core :as xpath])

(defn expr-or-nil
  [maybe-expr]
  (try
    (xpath/xp:compile maybe-expr)
    (catch org.xml.sax.SAXException e)
    (catch javax.xml.xpath.XPathException e)))

(s/def ::xpath (s/and string?           
                      (s/conformer (fn [value] {:value value
                                                :expr (expr-or-nil value)})
                                    :value)
                       #(:expr %))) 

(s/conform ::xpath "//a/@href")
;; {:value "//a/@href",
;;  :expr
;;  #object[org.apache.xpath.jaxp.XPathExpressionImpl 0x1aa56eab "
I do similar things when I accept regex strings as input - I validate by compiling them with re-pattern in a conformer then I save the output from s/conform for use later. So is it possible to do something like this with Malli? The :decode/math examples in the Value Transformation section of the README.md (https://github.com/metosin/malli#value-transformation) seem possibly close? But it's not clear to me if decode validates. Maybe Malli is trying to keep validation in a different stream from transformation? (I admit I don't understand the transformation section of the docs very well, particularly when it comes to doing custom functions as opposed to the pre-canned transformers for json and so forth.) Kiitos/thanks for any help
#2022-03-0906:32ikitommi@ryantate you have two models there: a string and a parsed map. With malli, you should describe either the string (source) or the target (a map). If the map presentation is what you expect, describe that and add a decoder from string->it. A naive impl:
(defn expr-or-nil [maybe-expr]
  (when (= maybe-expr "//a/@href") identity))

(defn decoder [value]
  {:value value
   :expr (expr-or-nil value)})

(def Xpath
  (m/schema
   [:map {:decode/string decoder}
    [:value :string]
    [:expr fn?]]))

(defn coerce [value]
  (let [decoded (m/decode Xpath value (mt/string-transformer))]
    (if-not (m/validate Xpath decoded)
      ::invalid
      decoded)))

(coerce "//a/@href")
; => {:value "//a/@href", :expr #object[clojure.core$identity 0x2c879e65 "
#2022-03-0906:35ikitommibut yes, the transform and validation are separated. There might be use cases where this is not good, but for most cases, it’s easy to compose the two using public apis. If/when someone finds out a need for one-pass-do-it-all &/ stateful transformers, would like to hear those 😎#2022-03-0913:38Ryan Tate@ikitommi Ah thanks for the comprehensive reply!! 👍 string -> compiled-xpath is the goal, the map is just there to allow s/unform which is not essential. In my case I would probably just accept the cost of two passes because I need the xpath compilation in order to validate the string) and also the xpath string needs to be composed into a more complicated validation so I can just do one m/validate call on the larger structure (which can have many xpath strings). So I think the coerce function would be tricky to use. I could just validate all xpath as string in the larger value and then m/validate and m/parse the larger value and then walk it and call coerce on each xpath item but I would have to do that also for regex and anything else that gets compiled - becomes basically building a new validation layer which defeats the point a bit. I did a benchmark and compiling xpath is only about 2 μs and regex is even less so I will just put those checks in the schema so it is part of a compostable validation and then do it again in m/decode. It is a performance hit but a small one, hopefully not too bad 😅#2022-03-0913:55ikitommioh, if you want to validate the big thing, you could do schema for string, and then m/encode it to the parsed format. there is no easy way to customize m/parse per schema atm.#2022-03-0913:56ikitommifor performance, you should use m/validator & m/decoder / m/encoder for MUCH better perf.#2022-03-0913:58ikitommi
;; pure functions, can be cached
(def validate (m/validator Xpath))
(def decode (m/decoder Xpath (mt/string-transformer)))

(defn coercer [value]
  (let [decoded (decode value)]
    (if-not (validate decoded)
      ::invalid
      decoded)))
#2022-03-0918:18Ryan TateIssue with :orn schema inside decode. Any idea why this works:
(m/decode  [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}] "foo" mt/string-transformer)
;;#"foo"
But this does not?:
(m/decode [:orn [:v vector?] [:re-string [:string {:decode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)
;;"foo"
Is decode meant to work with :orn, :catn, :or, :and etc?
#2022-03-0918:51Ryan TateActually I think I have mixed up encode and decode. I found this which has helpful definition of differences (maybe later I can make a PR to merge this table into README.md?): https://cljdoc.org/d/metosin/malli/0.8.4/doc/value-transformation decode is for making values valid, encode is to transform valid values into something else. decode will revert value if it does not match the schema per https://clojurians.slack.com/archives/CLDK6MFMK/p1615225531254000 I rewrote this way which works:
(m/encode [:orn [:v vector?] [:re-string [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}]]] "foo" mt/string-transformer)
;;#"foo"
#2022-03-1006:48ikitommiDoc PR most welcome#2022-03-1006:49ikitommiAlso, a transformation debugger would be nice (and not hard to implement?): would emit which steps are run, in which order and how they change / not the value.#2022-03-0919:13Ryan TateCan I combine parse and encode functionality using a single schema? I can parse:
(def re-schema [:string {:encode/string (fn [s] (try (re-pattern s) (catch java.util.regex.PatternSyntaxException e)))}])
(def re-w-opts-schema [:and vector? [:catn [:pattern re-schema] [:opts [:* [:enum :i :s :u]]]]])
(def re-maybe-w-opts [:orn [:pattern re-schema] [:pattern-w-opts re-w-opts-schema]])

(m/parse re-maybe-w-opts ["foo" :i])
;;[:pattern-w-opts {:pattern "foo", :opts [:i]}]
I can encode:
(m/encode re-maybe-w-opts ["foo" :i] mt/string-transformer)
;;[#"foo" :i]
How to do both and get
[:pattern-w-opts {:pattern #"foo", :opts [:i]}]
?
#2022-03-1007:00ikitommicurrently, you can’t combine those and there is no schema property-based extension for parsers, so you could plug in custom logic to parsing. I think you could cal m/parse in a custom encoder :leaveyourself?#2022-03-1007:03ikitommioh, the branching happens at parse, would not work that way. should add hook to parsing to allow custom steps or somehow mix the two (encode & parse). ideas welcome#2022-03-1415:31Ryan TateThanks for the reply, going to work on this for a bit. Parsing is more important for my use case than other transforms. Will come back if I have API extension ideas. Kiitos 👍#2022-03-1921:37Phil JacksonI came to ask exactly the same thing. @ryantate can you let me know if you come up with something?#2022-03-1013:40CélioMaybe I’m doing it wrong, but is there a reason why transformers don’t handle enums?#2022-03-1013:40CélioAn example with malli.transform/json-transformer:
(require '[malli.core :as m])
(require '[malli.transform :as mt])

(def schema [:enum {:encode name :decode keyword} :a :b :c])

(m/encode schema :x mt/json-transformer)
;; => :x

(m/decode schema "x" mt/json-transformer)
;; => "x"
Then I created a custom json transformer based on malli.transform/json-transformer where it adds codecs for :enum:
(defn my-json-transformer
  ...
  (mt/transformer
    {:name :json
     :decoders (-> (mt/-json-decoders)
                   ...
                   (assoc :enum mt/-string->keyword))
     :encoders (-> (mt/-json-encoders)
                   (assoc :enum m/-keyword->string))}))

(m/encode schema :x my-json-transformer)
;; => "x"

(m/decode schema "x" my-json-transformer)
;; => :x
#2022-03-1014:09Ben SlessTry decode/json ?#2022-03-1016:36CélioThanks, that works.#2022-03-1021:41DiegoHello everyone. What’s the best way to use malli for generative testing?#2022-03-1212:17Felipeif I’m not mistaken malli’s generators are compatible with clojure.test.check, which is what I’d use https://github.com/clojure/test.check#2022-03-1218:07DiegoThanks @UA2U3KW0L I’ll try that.#2022-03-1211:17Adam Helins@andres.rodriguezhn Edit: sorry, I was struggling with a situation similar to https://github.com/seancorfield/build-clj/issues/7 regarding tools.build and setting a custom registry. I eventually managed, thanks for raising attention on this :thumbsup:#2022-03-1309:08dharriganWhich way is the direction going re: vector vs map syntax?#2022-03-1309:08dharriganIs map the way forward?#2022-03-1311:16Ben SlessI'm guessing vector syntax will never go away due to backwards compatibility#2022-03-1314:43valtteriYou can pick whichever you prefer.#2022-03-1314:44valtteriIf I recall correctly, Tommi mentioned that map syntax is basically sugar over vector syntax#2022-03-1314:55valtteriAch, sorry it was the other way around. Map syntax is closer to the internal syntax.#2022-03-1314:57dharrigan:thumbsup:#2022-03-1413:51Martynas MHey. What function should I use to parse a string into a number? Let's say I have this schema: (m/parse :int 15) My initial assumption was that the parse would do the string parsing. But it... doesn't work at all. How do I make this return 15 : (m/parse :int "15") Is this the right function to use?#2022-03-1414:06ikitommihi, it’s value transformation in malli, here are the docs: https://github.com/metosin/malli#value-transformation#2022-03-1415:46Martynas MThanks. That works.#2022-03-1415:47Martynas MWhy does parse exist then?#2022-03-1416:40ikitommiit's structural parsing, please read the part from the README. Malli itself uses it for parsing Clojure destructuring syntax to infer schemas from source code.#2022-03-1417:21Ryan TateIs it possible to combine :and with :cat? In spec this would be "`&`" (ampersand). I need to check each key with :cat and then the two keys together via a special fn. This works without the :and but once I add the :and and the :fn it fails:
(m/validate [:* [:and [:cat :symbol :string] [:fn (fn [[key value]] true)]]]  '(regex "foo"))
Ultimate form is going to be like
'(regex "foo" xpath "bar" ...) 
and I want :fn to check each pair. Thanks for any tips.
#2022-03-1417:36Ryan TateAck I don't think it will work anyway since the :cat does not produce the values for the next predicate as in spec. So even if I could make :and work it's not gonna give key/value to :fn. Hmm. Not sure how I can accomplish this. Maybe I can parse inside the validation#2022-03-1417:43Ryan TateOK I solved it like this, the special function is the inner one:
(m/validate [:fn (fn [seqn] (every? (fn [[key value]]  true) (m/parse [:* [:cat :symbol :string]] seqn)))] '(regex "foo" xpath "bar"))
👍 If anyone has a better idea just post here 🙃
#2022-03-1417:49Ryan TateIn order to make it work and composable with m/parse I think the schema will need to repeat the seqex. So parse will run 2x. Probably will still be faster than spec:
[:and [:fn (fn [seqn] (every? (fn [[key value]] true) (m/parse [:* [:cat :symbol :string]] seqn)))] [:* [:cat :symbol :string]]]
#2022-03-1417:59Ryan TateWould still be nice if :and worked with cat though because then I could match multiple schemas across the same sequence. hmm#2022-03-1707:45rovanionDoes Malli happen to have a shorthand for making a map out of already defined value-specs like clojure.spec's (s/keys :req-un [:thing/omabob])?#2022-03-1707:52ikitommi
[:map {:registry {"kikka" :int, "kukka" :string}} "kikka" "kukka"]
#2022-03-1707:53ikitommie.g. if you have a schema in a registry, you can use the name instead of full entry definition.#2022-03-1708:30rovanionRight, thanks. And if I wanted to make the key in the map unqualified while the entry in the registry is qualified by a namespace, any way around that?#2022-03-1708:31rovanionWell, [:key :ns/key] works of course.#2022-03-1710:22ikitommiyes, and you can make add entry props too: [:map [::id {:optional true}]].#2022-03-1710:23ikitomminot sure was adding all that sugar a good idea, made the parser more complex - and slower#2022-03-1713:52rovanionKey/entry props I do use, so I appreciate them.#2022-03-1713:57rovanionIs it possible to add properties to an already existing custom type?
(def registry                                                                                               
  (atom {}))

(defn register! [type ?schema]                                                                              
  (swap! registry assoc type ?schema))

;; Combine the default registry with our own mutable registry.                                              
(mreg/set-default-registry!                                                                                 
 (mreg/composite-registry                                                                                   
    (mreg/fast-registry (malli/default-schemas))                                                            
    (mreg/mutable-registry registry)))


(register! :non-empty-string [:string {:min 1}])
(malli/validate :non-empty-string "Bengt")
;; => true                                                                                                  
                                                                                                            
(malli/validate [:map [:namn :non-empty-string]] {:namn "Bengt"})
;; => true                                                                                                  
(malli/validate [:map [:namn [:non-empty-string {:max 2}]]] {:namn "Bengt"})
;; Throws:                                                                                                  
;; Execution error (IllegalArgumentException) at malli.core/eval15223$fn$G (core.cljc:22).                  
;; No implementation of method: :-into-schema of protocol: #'malli.core/IntoSchema found for class: clojure\
.lang.PersistentVector                                                                                      

;; While this works:                                                                                        
(malli/validate [:map [:namn [:string {:max 2}]]] {:namn "Bengt"})
#2022-03-1716:58ikitommicurrently no, but would be a ~1 line change, haven’t thought of that use case… please write an issue,#2022-03-1902:39vinurshello, i define a date schema, validate it is false, what the correct format is ?
(def birthday
  [:fn {:error/message "wrong birthday format" :description "birthday"}
   (partial instance? java.time.LocalDate)])
(m/validate birthday "1987-09-06") => false
#2022-03-1908:36Ben SlessYou need to add a decoder and decode before validation#2022-03-2018:50ikitommi
((partial instance? java.time.LocalDate) "1987-09-06")
; => false
#2022-03-1919:55esp1Quick question: what is the purpose of having equivalent predicate and type schemas, e.g. int? and :int? Is it simply for convenience? Also, i'm noticing that when I validate using :vector I get an error, while when I use vector? it works:
(m/validate :vector [1 2 3])
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
; :malli.core/child-error {:type :vector, :properties nil, :children nil, :min 1, :max 1}
(m/validate vector? [1 2 3])
true
#2022-03-1919:56esp1this is with malli 0.8.4#2022-03-2018:48ikitommi> Is it simply for convenience? yes, for people coming from spec. the predicates will be made optional in the future,#2022-03-2018:49ikitommicurrently, the :vector expects one child. “vector of anything” would be [:vector :any]#2022-03-2101:17steveb8nAlso useful if persisting specs to a db which is sometimes useful#2022-03-2200:28esp1cool, thanks for the explanation!#2022-03-2116:06Alex SkyHello! Is it possible to use malli with compojure + swagger? Couldn’t find any examples. I can only see an option to implement the protocol Coercion#2022-03-2116:32ikitommiSadly no. Reitit has built-in support for malli#2022-03-2205:45Alex SkyIs it related to some kind of restrictions? I’m a little surprised since compojure-api as well retitit and malli are created by metosin :)#2022-03-2206:11bedersI have a hack to turn a malli spec into a Schema, if that helps#2022-03-2206:14Alex Sky> I have a hack to turn a malli spec into a Schema, if that helps Yes, could you show me an example?#2022-03-2206:20bedersI haven’t tested that on a compojure-api route yet, but here’s the gist of it:
(require '[schema.core                   :as schema]
         '[schema.spec.core              :as schema.spec]
         '[schema.spec.leaf              :as leaf])

(defn malli->schema [malli-schema]
  (reify schema/Schema
    (spec [this]
      (leaf/leaf-spec
       (schema.spec/precondition this
                                 (fn [data]
                                   (m/validate malli-schema data))
                                 (fn [data]
                                   (me/humanize (m/explain malli-schema data))))))
    (explain [this]
     (str "Malli->Schema:" malli-schema))))
#2022-03-2206:21bedersbut this allows you to use schema like this:
(schema/check 
 (malli->schema [:map [:foo int?] [:bar keyword?]])
 {:foo 1 :bar 2})
=> (not {:bar ["should be a keyword"]})

(schema/check 
 (malli->schema [:map [:foo int?] [:bar keyword?]])
 {:foo 1 :bar :humbug})
#2022-03-2206:23bedersnot perfect, and I’m about to use this in :body and :path-params and see if that works#2022-03-2206:56Alex SkyThanks!#2022-03-2122:42Godwin Kois it possible to get the current index of a vector spec, so that I can have different validation logic for the exact nth element?#2022-03-2206:40ikitommino, but use can use sequence schemas for that#2022-03-2301:02Godwin Kook, but it’s a bit clumsy if the validation of a specific item share most but just varies a little bit from the other…… thx a lot for your prompt respond anyway :man-bowing::skin-tone-2:#2022-03-2305:32ikitommiyou can also compose with [:and [:vector :int] [:fn (fn [v] (= (nth v 5) 42))]]#2022-03-2305:56Godwin Kogot it, i.e. raise one level to apply function spec on the entire vector :thinking_face:#2022-03-2305:59Godwin Koin our use case, we have vector of maps, using function spec to replace the entire map schema still not that straight forward or desirable…… 😅#2022-03-2212:19Nikolas PafitisI have this schema
(def ResourceIdentifier
  [:schema {:registry
            {::resource-key   :keyword
             ::path-param     [:or :string :int]
             ::query-params   map?
             ::resource-ident [:cat
                               [:+ [:or
                                    [:ref ::path-param]
                                    [:ref ::resource-key]]]
                               [:? [:ref ::query-params]]]}}
   ::resource-ident])
and i get a :malli.core/potentially-recursive-seqex although I don't see how it's recursive.
#2022-03-2212:30ikitommiyou should take away the :ref wrapping. :ref is potentially recursive. Using a plain reference value (e.g. ::path-param) should work ok.#2022-03-2310:36Nikolas Pafitis@U055NJ5CC I see, thanks alot.#2022-03-2212:25Nikolas PafitisThe equivalent in clojure.spec works fine
(s/def ::resource-key keyword?)
(s/def ::path-param (s/or :string string? :int int?))
(s/def ::query-params map?)
(s/def ::resource-ident (s/cat :path (s/+ (s/or :resource-key ::resource-key
                                                :path-param ::path-param))
                               :query-params (s/? ::query-params)))
#2022-03-2311:35CarloI'm looking into trying malli for my next project; is there some generative testing facility (I can't find it). How do you do generative testing?#2022-03-2311:36dharrigan#2022-03-2311:41Carlothanks @dharrigan, so to check that a function respects the malli spec you have your own thing that generates arbitrary input data and then calls the function?#2022-03-2311:42ikitommiwhat about: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defn-checking#2022-03-2311:43ikitommiIf there are some evident tools missing (I think there are), please tell 🙂#2022-03-2311:43Carloyes thanks, I don't know how I missed this file while googling 😍#2022-03-2311:44ikitommialso, you can get a test.check generator out of a schema with (malli.generator/generator my-schema).#2022-03-2401:32Carlotwo follow-up questions: is there a built-in way to (mi/check) only one function? Is there a way to pretty-print the result of (mi/check)?#2022-03-2410:14CarloTo check just one function from emacs, I ended up doing:
(defun med/cider-eval-on-top-level-form (fn-str)
  (let ((quoted-defn (concat "'" (cider-defun-at-point))))
    (cider-interactive-eval (concat "(" fn-str " " quoted-defn ")"))))

(defun malli-check-this ()
  (interactive)
  (med/cider-eval-on-top-level-form
   "#(mi/check {:filters [(mi/-filter-var #{(resolve (second %))})]})"))
#2022-03-2412:38CarloAbout the (mi/check) pretty-printing issue, I ended up coding my visualization for #portal that looks like this (it happens when I issue the mi/check command, that I bound to a key):#2022-03-2413:41ikitommiLook great! There is malli.dev.pretty for pretty printing things. Could add a handler for check there#2022-03-2413:50CarloThank you! Mostly, the question I had while doing this was, is there a malli spec for the kind of errors that are generated by (mi/check)? I temporarily shimmed it as:
(def check-error
  (m/schema
   [:map-of symbol? [:map
                     [:errors [:sequential
                               [:map
                                [:check [:map
                                         [:malli.generator/explain-output :any]]]]]]]])) 
but maybe you have something more complete. Btw, I'm really liking malli, it's really well designed! 😍
#2022-03-2414:22ikitommithanks! there is no schema for it, but could be, PR welcome.#2022-03-2416:25rovanionIs the following untrue because what's compared is just the addresses in the references?
(deftest optionalize-db-generated-keys
  (= (malli.util/optional-keys [:map [:key [:int]]])
     (malli/schema [:map [:key {:optional true} [:int]]])))
#2022-03-2416:54ikitommiRecall there is mu/equals, using just form checking. Could be improved...#2022-03-2416:42CarloI asked this quite some time ago, and IIRC at the time it wasn't possible: what's the way of writing a spec for the function (defn add [x y] ...) so that we check that the result is bigger than both x and y?#2022-03-2416:53Carlooh yeah I found a relevant issue https://github.com/metosin/malli/issues/608#2022-03-2417:11Carlo@U055NJ5CC I see that https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2443-L2445 is the place in which the function is instrumented, and that's probably where a check for the new :fn key of :=> should go. But what else should be changed?#2022-03-2501:21ambrosebsI recently added static type checking for malli schemas to Typed Clojure, here's a writeup on how to do it https://github.com/typedclojure/typedclojure/tree/main/example-projects/malli-type-providers
#2022-03-2507:14Karol WójcikOh Yeah!#2022-03-2510:09Ben SlessHow does it interface with custom registries and custom schemas?#2022-03-2511:42juhoteperiCool! What is the tooling like nowadays? Does it require REPL (so it is one the same classpath as the running app) or could it be integrated into clojure-lsp?#2022-03-2514:37ambrosebs@UK0810AQ2 it's a https://github.com/typedclojure/typedclojure/blob/4818f27a4661262b0d8268a6b00bd0363b02dba0/typed/malli/src/typed/malli/parse_type.clj#L196 right now for custom schemas, a big case. Could probably be a multimethod, do you have an example of a custom schema to help me understand the design space? And custom registries, I'm not sure how far it gets me, but I use m/-deref when I find a :ref. That should be enough?#2022-03-2514:38ambrosebs@U061V0GG2 requires a REPL since it macroexpands code.#2022-03-2514:39ambrosebsMaybe if we create a custom macro rule for every macro in clojure.core we can think of a clojure-lsp version.#2022-03-2514:39Ben SlessA good example of custom schemas would be for time types, such as https://github.com/metosin/malli/pull/545 Similarly for other iterop-y parse-y things, like valid URIs#2022-03-2514:44ambrosebsok, I think we need something like (defmethod malli->Type :local-date [_] 'java.time.LocalTime) .#2022-03-2514:45ambrosebsMaybe if the :class were propagated to the schema itself it would be even simpler.#2022-03-2514:50Ben SlessSo if those schemas had a way to query their backing class it would make it easier to participate in typed.clojure?#2022-03-2514:52ambrosebsin the simplest cases, yes. Is a "simple schema" usually tied to a class?#2022-03-2514:53Ben SlessNo, but I could petition for something like "simple class schema" which could back various boring data classes#2022-03-2514:55Ben SlessAnd maybe add in malli a protocol for accessing those#2022-03-2514:55ambrosebsthat sounds perfect.#2022-03-2515:00ambrosebsCan you think of any custom registry case that might be problematic? I just call m/deref to convert the entire schema eagerly.#2022-03-2515:01Ben SlessRecursive schemas?#2022-03-2515:02ambrosebsah yep, I handle those https://github.com/typedclojure/typedclojure/blob/4818f27a4661262b0d8268a6b00bd0363b02dba0/typed/malli/src/typed/malli/parse_type.clj#L236#2022-03-2515:03Ben SlessShould be fine How about dependent schemas?#2022-03-2515:03ambrosebsExample?#2022-03-2515:05Ben Slesshttps://github.com/metosin/malli#content-dependent-simple-schema#2022-03-2515:07ambrosebsprobably would need a custom m/type dispatch in the (future) malli->Type multimethod.#2022-03-2515:10ambrosebsanother way these simple schemas could be automated is if Typed Clojure could know the var backing the :pred. eg., pos-int? here#2022-03-2515:11ambrosebsbecause the checker has the annotations for these preds#2022-03-2515:12ambrosebsbut there's a lot of info in the json-schema props too.#2022-03-2515:16ambrosebsperhaps a :type-properties :typedclojure/type convention might be useful too.#2022-03-2517:23ikitommithis is great 🙂#2022-03-2521:41ambrosebsthanks Tommi 🙂#2022-03-2517:19ikitommiwelcome .pretty/prettifier, thanks @meditans for the idea. Using it to create pretty checker:
(require '[malli.instrument :as mi])
(require '[malli.dev.pretty :as pretty])

(defn check
  "check that emits pretty results"
  ([] (check nil))
  ([options] ((pretty/prettifier ::check "Check Error" (fn [] (mi/check)) options))))
Dummy pretty printer for ::check:
(defmethod v/-format ::check [_ _ results printer]
  {:body (->> (for [[f error] results
                    :let [{:keys [schema errors]} error
                          explanation (-> errors first :check :malli.generator/explain-output)]]
                [:group
                 (pretty/-block "When calling:" (v/-visit (-> errors first :check :smallest first (conj f)) printer) printer) :break :break
                 (pretty/-block "We get:" (v/-visit (:value explanation) printer) printer) :break :break
                 (pretty/-block "The problem is that:" (v/-visit (me/humanize explanation) printer) printer) :break :break
                 (pretty/-block "Schema:" (v/-visit schema printer) printer)])
              (interpose [:group (v/-color :title-dark (apply str (take 30 (repeat "-"))) printer) :break :break])
              (into [:group]))})
in action:
(check) 
#2022-03-2517:21ikitommithe actual printer is silly here, should separate the input & output schema problems, looking forward to seeing what you @meditans cooked up for this.#2022-03-2517:23ikitommibut, the core-lib support pretty-anything now. the helper looks like:
(defn prettifier [type title f options]
  (let [printer (assoc (or (::printer options) (assoc (-printer) :width 60)) :title title)
        actor (::actor options reporter)]
    (fn [& args] (when-let [res (apply f args)] ((actor printer) type res) res))))
#2022-03-2519:45CarloI am in fact still at a prototype stage re: precise messages (and I mainly produce hiccup to pass to #portal, which I'm trying to turn in my repl for visualization). But I'll definitely try this out and upstream improvements if I end up making them for myself. I have to say, ideally for me, this prettified error message would be more usable in the form of a data structure (from which I can generate hiccup). But I can as well keep producing it myself from the error!#2022-03-2520:33ikitommiI think a shared function of check-explanation to some intermediate map would be good and could be used by both the malli pretty-printer and your portal visualizer.#2022-03-2615:47CarloI got an error about trying to import malli.instrument in cljs, and in fact I see that in the malli repo we have only a instrument.clj file. Does this mean that I can't do instrumentation in cljs?#2022-03-2620:41ambrosebshave you tried malli.instrument.cljs? It's apparently seen some bugfixes since the last release so I don't know if it works, but I happened upon it a few days ago https://github.com/metosin/malli/blob/master/src/malli/instrument/cljs.clj#2022-03-2620:56CarloThank you, you're right! Here's how I missed that: I looked at this folder https://github.com/metosin/malli/tree/master/src/malli and I could only find the clj version. So I wrongly assumed that was the only available possibility!#2022-03-2620:57CarloHow does the naming schema in the instrument folder work? I have never seen something like this!#2022-03-2702:28ambrosebsI'm guessing that malli.instrument contains functions for the clojure impl. the cljs impl must be implemented via macros (but otherwise with the same names), so a new namespace malli.instrument.cljs was created.#2022-03-2720:19CarloI'm still getting this at times, though:
------ REPL Error while processing ---------------------------------------------
(ns couperin.user
  (:require [portal.web :as p]
            [malli.core :as m]
            [malli.instrument :as mi]
            [malli.dev :as dev]))
The required namespace "malli.instrument" is not available, it was required by "couperin/user.cljs".
"malli/instrument.clj" was found on the classpath. Maybe this library only supports CLJ?
#2022-03-2720:22CarloSame thing for the namespace#2022-03-2720:25Carloah it's probably a weird interaction with shadow-cljs https://clojureverse.org/t/problem-using-malli-clojurescript-instrumentation-and-shadow-cljs/8612/3#2022-03-2803:42dvingohey, I contributed the cljs support, it's in need of some documentation. When using the instrumentation for cljs the namespaces should be: [malli.instrument.cljs :as mi] [malli.dev.cljs :as md] and there is kondo support but it must be executed at runtime (because the schemas aren't available during compilation (macroexpansion)) and prints to the console, with the intention being you copy it to a kondo config file. https://github.com/metosin/malli/blob/400dc0c79805028a6d85413086d4d6d627231940/src/malli/clj_kondo.cljc#L203 the js console errors need some work (the pretty one is designed with terminal emulators in mind, not js web console) but if you execute an instrumented fn via nrepl in an editor the error output there is better.#2022-03-2812:02CarloThank you @U051V5LLP, I'm a bit confused on how instrumentation works on cljs. Even after importing:
(:require [portal.web :as p]
            [malli.core :as m]
            [malli.instrument.cljs :as mi]
            [malli.dev.cljs :as dev])
doing
(comment
  (dev/start!))
and adding:
(m/=> -add [:=> [:cat :int :int] :int])
(defn -add [x y]
  "abcd")
I can still execute calls like:
(-add 1 "a")
getting "abcd" as my answer (and I see that I have the instrumentation message in the browser console:
..instrumented 
{ns: 'couperin.user', name: '-add', str: 'couperin.user/-add', _hash: 150786855, _meta: null, …}
just not any error whatsoever
#2022-03-2812:28CarloOk, if I call (dev/start!) just before a call to -add, then I get the errors, but if I then modify the definition of -add , the instrumentation is not redone automatically#2022-03-2813:44dvingoinstrumentation works by replacing the function implementations so when hot reload happens the original functions will be replace the instrumented code. https://shadow-cljs.github.io/docs/UsersGuide.html#_lifecycle_hooks you'll want to call instrument! or start! after the hot code reload runs (usually something like (defn ^:dev/after-load refresh [] (md/start!)...) for shadow.cljs#2022-03-2814:08CarloThank you, would you still advice to do https://clojureverse.org/t/problem-using-malli-clojurescript-instrumentation-and-shadow-cljs/8612/2 ? I can get it to check something but it's clearly confused by namespaces. Is there a particular place the refresh function you just mentioned should live?#2022-03-2900:22dvingojust the standard setup you'd have for any cljs (utilizing react) app: https://github.com/day8/re-frame/blob/69cf39552715fa410e7007b7fcbc894097d8db1f/examples/todomvc/src/todomvc/core.cljs#L57#2022-03-2711:20Ivan FedorovNew pretty explainer is so awesome! Great work, thanks a lot!#2022-03-2711:21Ivan Fedorovcan we expect it in cljs land?#2022-03-2813:31Carlothe new pretty explainer seems to break cljs instrumentation, because of not found variables.#2022-03-2816:17rovanionI am either missing something or I've found a bug. It seems like updating a key in a map-spec makes the the optional property have no effect, even though it is still there:
;;; Specification registry

(def registry
  (atom {}))

(defn register! [type ?schema]
  (swap! registry assoc type ?schema))

;; Combine the default registry with our own mutable registry.
(mreg/set-default-registry!
 (mreg/composite-registry
    (mreg/fast-registry (malli/default-schemas))
    (mreg/mutable-registry registry)))

(register! :db/f
   [:map
    [:key   {:optional true} [:double]]])

(register! :user/f
   (-> (malli/deref :db/f)
       (mutil/update :key #(mutil/update-properties % assoc :disabled true))))

(malli/deref :db/f)
;; => [:map [:key {:optional true} :double]]

(malli/deref :user/f)
;; => [:map [:key {:optional true} [:double {:disabled true}]]]

(malli/explain :db/f {})
;; => nil

(malli/explain :user/f {})
;; => {:schema :user/f, :value {}, :errors ({:path [0 :key], :in [:key], :schema [:map [:key {:optional true} [:double {:disabled true}]]], :value nil, :type :malli.core/missing-key})}

(mgen/sample :db/f)
;; => ({} {} {} {} {:key -1.5} {} {} {:key -1.09375} {:key -2.0} {:key 4.625})

(mgen/sample :user/f)
;; => ({:key -2.0} {:key -0.5} {:key 2.0} {:key -0.75} {:key -1.5} {:key 1.75} {:key 3.53125} {:key 1.25} {:key -0.25} {:key 0.271484375})
Yup, another user already found it: https://github.com/metosin/malli/issues/645
#2022-03-3013:25CarloI'm really liking the pretty explainer namespace, @ikitommi! Do you think this function https://github.com/metosin/malli/blob/d69e06326662c675a40095c7c72ca80af8a8d282/src/malli/dev/pretty.cljc#L69-L75 could get some more inputs? Like, the name of the function that's displaying the error, maybe the line location? Or is it a thing I should try to get via other means?#2022-03-3112:04CarloHere's a tentative PR for this feature @ikitommi, I'd love to know if this is the right direction and what could be improved https://github.com/metosin/malli/pull/680#2022-04-1310:18ikitommiwill look into this.#2022-03-3102:20mafcocincoIs it possible to supply an arbitrary generator function to an :fn schema (or any schema really)? I know this is trivial, but works for an example:
(malli.g/generate [:map [:a [:fn {:gen/function (fn [] 1)} #(int? %)]]])
#2022-04-1310:17ikitommitry :gen/gen.#2022-03-3102:20mafcocincomalli throws an exception with that example, so obviously not correct but was hoping there was something like that.#2022-03-3102:22mafcocincoThe reason I ask is that we have, for example, a customer schema with first-name, last-name, etc. The schema for first-name uses string? to validate but would like to constrain the generated values to a handful of simple names rather than the universe of all possible strings.#2022-03-3102:22mafcocincoWould definitely make the generated data a bit more readable and user friendly.#2022-04-0310:29CarloQuick question: does generative checking work in cljs? If not, why not? If I try to write (mi/check) I get:
------ WARNING #1 - :undeclared-var --------------------------------------------
 File: /home/carlo/code/clojure/visualizerTestCljs/src/main/core.cljs:12:1
--------------------------------------------------------------------------------
   9 | (defn badd [x y]
  10 |   "z")
  11 | 
  12 | (mi/check)
-------^------------------------------------------------------------------------
 Use of undeclared Var malli.generator/check
--------------------------------------------------------------------------------
edit: solution in thread
#2022-04-0310:43CarloOk, it works if I manually include an import on malli.generator , even if I don't directly use the namespace. This is probably due to how shadow-cljs includes the files. I'm going to leave it here in case it's useful to someone else in the future.#2022-04-0400:50dvingoI made an github issue to fix this. you shouldn't have to require that ns - I'll fix it in the coming days#2022-04-0509:18Ferdinand BeyerIs there a recommendation when to call m/schema and when not? I want to define schemas in vars, would wrapping them in in m/schema improve performance?#2022-04-0509:26Ben SlessAlthough I worked on improving parsing performance, calling schema will help with that It will also ensure the same object is shared if the schema is used multiple times#2022-04-0512:02Ferdinand BeyerOK thanks. I’ve seen that I can get the form with m/form if I need to 😉#2022-04-0718:19respatializedIs there a way to negatively specify a map key? Like "this map cannot contain the :ident key?"#2022-04-0718:20respatializedI could do an :and schema with a :not but that seems clunky and was wondering if there's another way#2022-04-0721:26ambrosebsAFAIK think it's like spec in this respect. There are hacks around it but it's not supported, https://github.com/metosin/malli/blob/3599fbd7fe1eba96c35e5c39073397cba0984b6a/src/malli/core.cljc#L975-L982 is concerning presence, not absence.#2022-04-0721:26ambrosebsanother hack I remember from spec is using map-of's keys with a restricted spec for keys.#2022-04-0721:29ambrosebswhen I https://github.com/typedclojure/typedclojure/blob/bb973bce811c111a814aad4889aa2a083a5e761d/typed/malli/src/typed/malli/parse_type.cljc#L148-L151 a Typed Clojure => malli translation, the best I could find was {:closed true} to prevent all other keys.#2022-04-0721:29ambrosebsThat might be what you want actually.#2022-04-0721:40respatializedUnfortunately not, my use case is a data model like GeoJSON where it's open by default but some keys are reserved and cannot be used in certain contexts#2022-04-0807:43Ben SlessYou can require a map be closed in malli#2022-04-0718:20respatializedI could do an :and schema with a :not but that seems clunky and was wondering if there's another way#2022-04-0807:28Ferdinand BeyerWhat’s the preferred way of testing schema compliance in tests? Works but does not give helpful failure messages:
(is (m/validate ,,,))
Better, but not super intuitive:
(is (nil? (m/explain ,,,)))
Is there some other way that I’m missing?
#2022-04-0807:42Ben SlessInstrument functions and use generators#2022-04-1012:33Martynas MHey. There is a guy that asks for more transparency on what the library supports. I'm not too sure how to answer his question. This probably means that he'd expect to either be dismissed and pointed into README once again or then the README should be updated. https://github.com/metosin/malli/issues/652 https://www.reddit.com/r/Clojure/comments/tykor2/malli_schema_questions/#2022-04-1016:23ambrosebsI took a crack at answering.#2022-04-1015:11pithylessThere is a theory of documentation, that I first saw promoted by Jacob Kaplan-Moss (highly influential in Django documentation and later Heroku Developer Center), that divides documentation into 4 kinds: tutorials, how-to guides, reference and explanation. In general, when I see people struggling with documentation, it is often because the author of the document in question and the reader are in two different quadrants - neither is wrong, but they're missing each other instead of mind-melding. @invertisment_clojuria - having skimmed the linked issues, I think the same mismatching of expectations is happening. Malli does have some tutorial-like and explanation-like content in the README, and one can explore the codebase and tests to gather more reference-like content. https://malli.io gives examples one may consider how-to guides for modeling common schemas. But I think it is a fair assessment that all these things are scattered and not easy to find, especially for a newcomer to the library. Aside from Jacob Kaplan-Moss talks and writing you can find online, I found a good write up of the problems with structuring documentation here: https://documentation.divio.com/ I think moving forward, malli would benefit by taking some pointers for how to organize different kinds of documentation and information for the community. It's not a question of what should be in the README, tests, etc., but more of a question of how to organize this information that a reader coming to the project with different mindset and expectations can find the information that is most relevant for them at the time.#2022-04-1208:39eskosHi, did a bit of a hit&run yesterday with this, but wanted to comment 🙂 I totally agree with this! Documentation, in general, is both incredibly hard and incredibly valuable, and I do use that divio link every time I talk about software documentation to anyone as reference - I don’t claim it to be perfect, but it is definitely good enough. For me, malli’s documentation currently has a discoverability issue. Over the past few years malli has gained a daunting amount of features, but there’s no clear path to learning each feature, and the readme’s examples aren’t comprehensive - this latter is especially kinda amazing considering the readme is probably the longest I’ve ever seen in a Clojure project, especially when considering the amount of examples. To generalize what I want from documentation is three things: 1. How do I use this. This is always the most critical one, and a sort of blindfold approach - if there isn’t a direct “do x to get y, do z after doing y to get z” kind of How-To guide, I’ll probably never learn about That One Awesome Feature. This is sort of related to the eternal discussion of if left fold is enough or should we also have map and reduce, or is filter enough or do we need keep and remove as well. 2. What is the internal logic. I don’t mean algorithms or stuff like that, but why are things organized in the way they are and what are the architectural decisions. “We like interceptors over middleware” sounds nice, but that assumes I know what interceptors and middleware are. Another would be naming schemes used to name functions, especially prefixes and suffixes used, stuff like that, or something thankfully not that common in Clojure land, but do you return raw values, lazy seqs, promises, channels...how do you assume your library is being used from the outside? 3. All I need is the documentation. If I need to read the source to understand software behavior, the documentation sucks. I’ve been told especially about this that I am Wrong™, but I don’t have time nor interest for that debate - this is my approach 🙂 From technical point of view IMO malli should add one of those libraries which can evaluate code snippets in README just to make sure the snippets stay current and working. Sort of related but inevitable is also the fact that malli’s build is now build+deps based, which is like https://github.com/Gant/Gant all over again, but ehhhh pointing back to my previous points, if I need to worry about how malli is built, I’m already too deep… 🙂#2022-04-1212:04pithylessThanks for the long and insightful comment @U8SFC8HLP :) It's going to be a slow start, but I'm willing to try to help out with this kind of documentation in the coming weeks. So the core team can focus on delivering more awesome features. ;) I see Tommi is vacationing atm, but I wonder if other malli contributors or users have more insights on what they'd like to see. I'll start taking some notes.#2022-04-1310:16ikitommi@UK0810AQ2 , you had a tutorial almost done? PR would be most welcome. Also, @U05476190 & all, all documentation improvements are most welcome, /docs being a good place for them.#2022-04-1110:56Martynas MHey. Is there a way to specify that a schema decoder should decode into a list instead of a vector? I'd like to have this schema: [:list :string] Is this possible without additional code (be it in the transformation step or after the decoding)? I think a decoder could pick this one up but it would be a custom one: [:vector {:type :list} :string] But I simply want to not add any code at all, if possible (maybe to serialize the schema later, not sure). (type (malli.generator/generate [:sequential :nil])) => clojure.lang.PersistentVector#2022-04-1310:14ikitommicurrently, no. but would be nice. there is a collection-transformer but it’s current purpose is to ensure the right format, should read the :type etc. property for the correct target type#2022-04-1310:00Yehonathan SharvitHello, We are building an automatic form generator based on Malli. The form generator input is made of: • A malli schema • A structural description of the form The structure of the form is not necessarily the same as the structure of the data. Here is a example
{:schema [:map 
          [:username {:title "Username"} :string]
          [:email {:title "Email"
                   :description "The business email of the user"}
           :string]
          [:personal [:map 
                      [:age {:title "Age"
                             :description "The age of the user"} :int]]]]
 :ui {:sections [{:title "General"
                  :fields [{:path [:username]}]}
                 {:title "Details"
                  :fields [{:path [:personal :age]}]}]}}
Now, my question is: is there a way to retrieve the schema that corresponds to a map field? For instance the schema that correspond to the [:personal :age] field.
#2022-04-1310:07ikitommisure, but there can be many schemas behind a path, if there is an :and or :or. but if you know what you are doing, you can just the the first one, like this:
(defn schema-in [?schema path]
  (let [schema (m/schema ?schema)]
    (->> (mu/in->paths schema path) (first) (mu/get-in schema))))

(schema-in
 [:map
  [:username :string]
  [:email :string]
  [:personal [:map
              [:age :int]]]]
 [:personal :age])
; => :int
#2022-04-1310:09ikitommibtw, just doing the same thing in a projects (again), would like to push some parts back to the library#2022-04-1311:03Yehonathan SharvitWhat exactly are you doing in your project? UI generation?#2022-04-1312:35ikitommiyes, two cases: 1. technical ui’s for admin/prototyping, directly from malli schemas 2. having large amount of dynamic forms & rules in project(s) => need both utilities for malli-backed form components and some data-oriented generic form-markup, “the ui-schema” … looking at your example, you are doing 2 too and I know there are many others doing that, looking forward to see if there could be something reusable / shared with these.#2022-04-1316:15Yehonathan SharvitNice!#2022-04-1316:16Yehonathan SharvitHave you found a way to deal with :multi?#2022-04-1408:40Yehonathan Sharvit?#2022-04-1310:07ikitommisure, but there can be many schemas behind a path, if there is an :and or :or. but if you know what you are doing, you can just the the first one, like this:
(defn schema-in [?schema path]
  (let [schema (m/schema ?schema)]
    (->> (mu/in->paths schema path) (first) (mu/get-in schema))))

(schema-in
 [:map
  [:username :string]
  [:email :string]
  [:personal [:map
              [:age :int]]]]
 [:personal :age])
; => :int
#2022-04-1518:53hjrnunesHi. Say I want to validate a sequence of maps and make sure there's at least one of them with a specific entry. Is this in scope for Malli? How would I go about it? Example:
;; my schema is something that somehow looks for [:b 2] in a map sequence

(m/validate my-schema [{:a 1} {:b 2} {:c 3}])
=> true

(m/validate my-schema [{:a 1} {:b 2} {:b 2 :c 4} {:c 3}])
=> true

(m/validate my-schema [{:a 1} {:c 3}])
=> false
#2022-04-1519:00Noah Bogartyou can use https://cljdoc.org/d/metosin/malli/0.8.4/doc/readme#fn-schemas to validate "anything":
(def my-schema
  [:and
   [:sequence map?]
   [:fn (fn [s] (some #(= 2 (:b %)) s))]])
i haven't tried that but it should work
#2022-04-1520:03Luke JohnsonI’m sure this has been asked before, but is there a way to generate a value compared to the generated value of a key in the same map schema?
{:min-value 250  ;; generated
 :max-value 350} ;; always exactly 100 more than min
Looking through the documentation, I can’t find anything specific. https://github.com/metosin/malli/blob/master/docs/tips.md#dependent-string-schemas looks promising or maybe using :gen/fmap but I could really use some guidance.
#2022-04-1614:14Ivan FedorovCan I define registry props with options and then reuse these options when I reference props in :map definitions?#2022-04-1817:35mauricio.szaboHi, I was playing with instrumentation on Malli, but I can't find a way to define a function that accepts two args: one int? and other a tuple of int? and string?. Essentially, I want to allow a function to accept (some-function 10 [20 "string"]) as arguments. How do I do this?#2022-04-1818:20ikitommimaybe:
(m/validate [:cat int? [:tuple int? string?]] [10 [20 "string"]])
; => true
#2022-04-1914:23mauricio.szaboYeah, that worked, thanks! I was trying with repeated :cat but that got me some weird messages......#2022-04-2108:28dharriganI'm wondering, after looking at the examples, I still can't really figure out how to decode a string into lowercase. I have something like this:#2022-04-2108:28dharrigan
(def foo [:map
            {:closed true}
            [:name [:string {:max 256 :error/message "should be at most 256 characters"}]]
            [:email-address [:string {:decode/string 'clojure.string/lower-case :min 5 :max 100 :error/message "should be between 5 and 100 characters"}]]])

  (m/validate foo {:name "foo" :email-address "
#2022-04-2108:29dharriganWhat my objective is (in tandem with reitit), is to ensure that the email address coming in is all in lowercase. Bit of a puzzler.#2022-04-2108:29dharriganAny pointers would be appreciated. Thank you.#2022-04-2109:05dharriganSo, got it to work, after plugging away at it for a bit:#2022-04-2109:05dharrigan(m/decode signup-request {:name "foo" :email-address "#2022-04-2109:06dharrigandifference being to use a string-transformer not a json-transformer#2022-04-2109:18dharriganAlthough, it doesn't appear to work with reitit and it's use of json-transformer.#2022-04-2108:51Roee MazorHi, I am looking at this example:
(m/validate [:alt keyword? string?] ["foo"]) ; => true
and I am wondering why it works that way, why not:
(m/validate [:alt keyword? string?] "foo") ; => true
#2022-04-2108:52Roee Mazor(the second one says false which means the alt thing actually makes it look for a sequence for some reason, how can I use :alt without a vec/seq?)#2022-04-2109:01Martynas MTry :or#2022-04-2109:02Roee Mazorperfect, that works perfectly ❤️#2022-04-2113:08Ben SlessAlt is for sequence schemas. It's the | operator in regular expressions#2022-04-2403:28minosniuHi, I'm getting a strange error when running the http://malli.io example of "Pet" in Clojure (not Clojurescript). Code here:#2022-04-2403:29minosniu
(def PetMalli
    [:schema {:registry {"Pet" [:map
                                [:type keyword?]
                                [:name string?]]
                         "Cat" [:merge
                                "Pet"
                                [:map
                                 [:type [:= "Cat"]]
                                 [:huntingSkill [:enum {:description "The measured skill for hunting"}
                                                 :clueless, :lazy, :adventurous, :aggressive]]]]
                         "Dog" [:merge
                                "Pet"
                                [:map
                                 [:type [:= "Dog"]]
                                 [:packSize [:int {:min 0,
                                                   :default 0
                                                   :description "the size of the pack the dog is from"}]]]]}}
     [:multi {:dispatch :type} "Cat" "Dog"]])

(m/validate PetMalli {:type "Cat", :name "Viivi", :huntingSkill :adventurous})
#2022-04-2403:29minosniuMessage: Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136). :malli.core/invalid-schema {:schema :merge}#2022-04-2403:31minosniuCould anyone give me some pointers? We want to use :merge for some inheritance-like constraints in our schema, which is very similar to the "Pet" example.#2022-04-2403:58Ben SlessMerge is in another registry. You need to merge it into the registry you're creating. It's in malli util, I think#2022-04-2404:19minosniuMy guess, too. But the code might get somewhat cumbersome. It's strange that the same code from the Pet example simply won't work, notice that in the example only local registry was used.#2022-04-2404:25Ben SlessYeah that's something I'm still iffy about. Will wait for Tommi to chime in#2022-04-2513:28dvingoI don't think it's that misleading - the code is available here where the initial default registry is setup: https://github.com/metosin/malli.io/blob/master/src/malli/web.cljs#L28#2022-04-2513:58dvingolinking to these sources from the site is probably a good idea though#2022-04-2514:04minosniuWorked like a charm:dizzy:#2022-04-2513:29dvingoI think it's apparent that the local registry being used does not contain all of the schemas available (not just :merge)#2022-04-2514:15ikitommimaybe the util-schema types should be prefixed? e.g. :merge -> :util/merge to make the it more explicit that they are not part of the default registry? btw this works too:
(def Merge (mu/-merge))

(m/validate [Merge [:map [:x :int]] [:map [:y :int]]] {:x 1, :y 2}) ; => true
#2022-04-2514:31minosniu:util/merge seems more prominent and alerting.#2022-04-2608:29Martynas MHey. Is there a way to decode a string into a symbol?
(malli.core/decode [:enum 'identity]
                   "identity"
                   decode-transformer)
I want to parse JSON and I want to parse a string into a symbol but I want to force this exact value. I could define my own decoder and I did it in one other case but I don't think I want to define too many of them.
#2022-04-2608:31Martynas MProbably this is the way:
[:and :symbol [:enum 'identity]]
#2022-04-2610:17Ben SlessYou can specify your own decoder in the schema#2022-04-2610:18Ben SlessWhere you can just put the symbol decoder#2022-04-2608:58Martynas MThis fails to generate: (malli.generator/generate [:and :symbol [:enum 'my-symbol]]) Should I create an issue?#2022-04-2611:31ikitommioh, that’s an test.check impl detail, not sure how much we can do for it.#2022-04-2611:31ikitommie.g. when multiple constraints, first one is used for generation and the rest to filter that the first one generated valid values.#2022-04-2611:32ikitommiquick & dirty fix, override the generator:
(malli.generator/generate [:and {:gen/schema [:enum 'my-symbol]} :symbol [:enum 'my-symbol]])
#2022-04-2805:59Martynas MI made this function for myself:
(defn exact-symbol [sym]
  [:and {:gen/schema [:enum sym]} :symbol [:enum sym]])
I simply use the generator to validate that my schema is correct. If it generates then it is somewhat correct. My assumption is that if the generator can pick it up then it's a valid definition (not that the result is valid but the definition).
#2022-04-2713:33Ferdinand BeyerI might have found a bug Malli’s clj-kondo type config generation when using :re schemas. Instead of expecting a string matching a regular expression, clj-kondo expects an argument to be of regular expression type. Or did I misunderstand something?#2022-04-2715:17dvingocan you share the clj-kondo configuration for that function?#2022-04-2715:18dvingothe kondo code outputs :regex for :re schemas: https://github.com/metosin/malli/blob/a3db330eae029863e9be443cda3a2cafd8f61a33/src/malli/clj_kondo.cljc#L101 and I'm not certain but I think kondo expects a string: https://github.com/clj-kondo/clj-kondo/blob/d9fca2705863e3e604e004ccb942e0b3d2e268ec/src/clj_kondo/impl/types.clj#L20#2022-04-2715:49Ferdinand BeyerUnfortunately I deleted this example file already, but I can confirm that I get :regex in the config, and I agree that this should probably be :string#2022-04-2715:49Ferdinand BeyerHaving said that, I never worked with clj-kondo that deeply so I don’t know what ops/types it accepts#2022-04-2717:50dvingolooks like the :regex type in kondo expects a regex: https://github.com/clj-kondo/clj-kondo/blob/master/doc/types.md added this to the config:
$ cat .clj-kondo/config.edn
{:linters
 {:type-mismatch
  {:level :warning
   :namespaces {foo {foo {:arities {1 {:args [:regex] :ret :string}}}}}}}}
#2022-04-2717:50dvingoand then:
$ clj-kondo --lint - <<<'(ns bar (:require [foo :refer [foo]])) (foo "")'
<stdin>:1:45: warning: Expected: regular expression, received: string.
linting took 12ms, errors: 0, warnings: 1
$ clj-kondo --lint - <<<'(ns bar (:require [foo :refer [foo]])) (foo #"")'
linting took 11ms, errors: 0, warnings: 0
#2022-05-0207:35Ferdinand BeyerI created a PR with a tiny fix for this here: https://github.com/metosin/malli/pull/701#2022-05-0208:51ikitommiMerged, thanks!#2022-05-0208:55Ferdinand BeyerThat was fast, thanks 🙂#2022-04-2718:17Keith HouserWhy does humanize not have "Wrong!!" message when wrong value is in the first position?
({:forms (d),
  :human [["should be a" "should be b" "should be c"]]}
 {:forms (a d c),
  :human [nil ["should be a" "should be b" "should be c" "Wrong!!"]]})
#2022-05-0207:35Ferdinand BeyerI created a PR with a tiny fix for this here: https://github.com/metosin/malli/pull/701#2022-04-2817:56dumratWhy?
(m/validate 
   [:map
    [:type :keyword]
    [:name {:min 1 :max 128} string?]]
   {:type :ledger :name ""}) => true
#2022-04-2818:01dharriganTry this:#2022-04-2818:01dharrigan
(m/validate
  [:map
   [:type :keyword]
   [:name [:string {:min 1 :max 128}]]]
  {:type :ledger :name ""}) ;; false
#2022-04-2818:01dharrigan
(m/validate
  [:map
   [:type :keyword]
   [:name [:string {:min 1 :max 128}]]]
  {:type :ledger :name "foo"})  ;; true
#2022-04-2908:13rayatNeed help: (shadow) CLJS + Malli + console multiline printing weirdness - really cannot figure this out. More details in thread ⬇️ But TLDR: `pretty/reporter` is doing something really crazy, it prints every single character/"word" (or fipp document node, if I'm not mistaken) on new lines creating MASSIVE illegible walls of text in my console. #2022-04-2908:20ikitommiare you using the latest code from master? I believe that is fixed.#2022-04-2908:21rayat> I've also tried other reporters, including just straight console log/warn/error, but since pretty/reporter is the closest to working and the one whose intended output I want, I'll focus on that one. I’ve tried pretty/reporter, console.log/error/warn, pretty/thrower and have even copied and modified tons of the function call chain of pretty/reporter However: 1. With thrower and default -fail completely illegible schema failures are printed a. It is extremely "horizontal/squished" b. and more importantly even inspected in devtools repl, I cannot see the schema failure contents, just the data of the schema I wrote in the first place i. (I've expanded all the folding bits in devtool's custom formatted content for the error) 2. I also tried replacing some logic within pprint-document (more on that fn to follow) to console.warn instead, and got some progress somehow a. This is the first screenshot b. But as you can see, it looks like it's printing actual characters, instead of what I believe might be encodings for whitespace? 3. And with pretty/reporter, it looks like something about the whitespace is messed up, and so is also nearly illegible a. (second screenshot) b. The content does print, but it looks like each word boundary whitespace is printed as a new line, and most of the time, it's several empty newlines per. c. As I step through pprint-document, it looks like after the document goes through (serialize document) and the (eduction chain, the print call in (run! print) receives each individual word, instead of the lines that (I think) should have those words in them
(->> (serialize document)
          (eduction
            annotate-rights
            (annotate-begins options)
            (format-nodes options))
          (run! print))
a. I thought it might have been the println line after the bit I just posted, but it just (as expected) added an extra newline after all the words from the schema error/document had been newline-printed b. In the screenshot you'll see console.group being used from one my modifications, just for convenience against scrolling through massive number of lines in the console I'm really confused and stuck by this, and can't seem to find any mention of this in the malli docs, api browser, GitHub issues and slack search about this, or if anyone else has faced this. I've tried: 1. explicitly setting the print fn to be console.log/error/warn, even window.alert to see if there was some global pollution of my console var making it split several times per word 2. inspecting the contents of document as passed to format-nodes and pruning bits of the latter's cond conditions/code to only address :group, :text etc 3. opening a fresh new chrome profile with no extensions or anything 4. removing any and all scripts from our <head> like tracing/analytics tools Maybe it's something about my work stack or setup or something. I'm new to clojure though, and so I've never created any clojure projects of my own, let alone shadowcljs web-apps. If it is required tho, I can try looking into making a repro project Does
#2022-04-2908:21rayat@U055NJ5CC, I have [metosin.malli LATEST] in lein, do you mean a direct git dependency or something?#2022-04-2909:34rayatOh crazy, I only just found the https://github.com/metosin/malli/issues/675. Unfortunately at my work we use lein, which has no native git-based dependency support. I think I'd have an easier time justifying the introduction of malli itself versus lein plugin into our work codebase, or split our lein config into using tools.deps for dependencies. Is there If not, I understand. But then I ask if there's any recommendation you can give to get as much of the latest master's fixes on this bug? Thanks again.#2022-04-2909:34rayatOh crazy, I only just found the https://github.com/metosin/malli/issues/675. Unfortunately at my work we use lein, which has no native git-based dependency support. I think I'd have an easier time justifying the introduction of malli itself versus lein plugin into our work codebase, or split our lein config into using tools.deps for dependencies. Is there If not, I understand. But then I ask if there's any recommendation you can give to get as much of the latest master's fixes on this bug? Thanks again.#2022-04-2914:27dvingoyep very recent fix: https://github.com/metosin/malli/commit/d1a28f734e6fa3e2f669d5cb08ecfcd0558991de#2022-04-2914:58dvingobeing stuck on lein is unfortunate, I led a project off of lein to deps across multiple repos last year and know that pain. however, thanks to Tommi's excellent design of having the reporter be a function you provide, you can address this in your own codebase: https://github.com/metosin/malli/blob/a3db330eae029863e9be443cda3a2cafd8f61a33/src/malli/dev/cljs.cljc#L32 just copy this ns: https://github.com/metosin/malli/blob/master/src/malli/dev/pretty.cljc and this one: src/malli/dev/virhe.cljc to include the relevant fix: https://github.com/metosin/malli/commit/d1a28f734e6fa3e2f669d5cb08ecfcd0558991de#diff-78b0ed5a31a32de704a1f77385e434d91bc5a44a0f74d37bd313ea96a130d97cR27 it should be noted there were two even more recent fixes for cljs though. the implication is that things will work without those fixes only for :malli/schema metadata style function schemas but that should get you unblocked!#2022-04-2916:16rayatThanks so so much for the help! I've been trying to figure this out for like a week, too embarrassed in case it was like an extension or external script or some user error that was causing it - I didn't even think to check if the latest master/docs were released or not. Regarding your points: 1. facepalm I was trying to copy individual functions lol, can't believe I didn't think to just copy the relevant ns completely. Will do this post-haste. 2. Of those 2 recent fixes for cljs, it looks like a. https://github.com/metosin/malli/commit/f7b4e12d3fe3e2f60ff50057118308b7a3abe148 is about collecting metadata fn schemas, doesn't that mean it's a fix only for metadata style fn schemas? So wouldn't things work without this fix if I didn't use metadata style? ( the inverse of what you said) Lastly, since I'm here: 1. I'm actually using neither style, as I'm trying to use malli with helix defnc components. a. I'm directly using a wrapper over (-instrument {:schema ... :reporter ...} some-fn-arg-from-wrapper) i. Would that have some implications regarding any of the since-last-release cljs fixes? b. Without too many specifics, but in case helps to know why, i. defnc does not forward metadata to the expanded fn, so that route doesn't work, would have otherwise used that 1. (yet again, there's an unpublished recent change that added metadata forwarding lol - oh lein) ii. the output of the defnc macro could sometimes be an Fn , in which case => would work, but at other times it produces a js obj with a key render that has the equivalent Fn 1. This is just a detail of React, wherein the latter case of the obj.render is a special case handled automatically by React iii. so my wrapper just instruments the output of defnc with the provided schema for this special case, so it picks out the render key's val if it's an obj, or the fn itself if it's an fn, and then passes either to -instrument #2022-04-2917:27dvingofor 2. even though it's about collect it doesn't impact functionality - just developer experience (requires saving your code twice to see new fn schemas)#2022-04-2917:28dvingook cool, I think using -instrument directly should work - for getting around helix not forwarding meta you can structure your code such that the body of all helix components proxy to your own functions of props -> react element#2022-04-2917:28dvingoand then just instrument those#2022-05-0313:22Nikolas PafitisHi, I'm using mr/set-default-registry! and I'm getting the following error when running kaocha test runner and the namespaces reload on change.
Caused by: java.lang.IllegalArgumentException: No implementation of method: :-schema of protocol: #'malli.registry/Registry found for class: nil
 at clojure.core$_cache_protocol_fn.invokeStatic (core_deftype.clj:584)
    ...
    malli.registry$eval15830$fn__15831$G__15819__15838.invoke (registry.cljc:11)
    malli.registry$custom_default_registry$reify__15870._schema (registry.cljc:50)
    malli.core$_lookup.invokeStatic (core.cljc:260)
    malli.core$_lookup.invoke (core.cljc:258)
    malli.core$_lookup_BANG_.invokeStatic (core.cljc:265)
    malli.core$_lookup_BANG_.invoke (core.cljc:263)
    malli.core$schema.invokeStatic (core.cljc:1985)
    malli.core$schema.invoke (core.cljc:1963)
    malli.core$properties.invokeStatic (core.cljc:1999)
    malli.core$properties.invoke (core.cljc:1994)
    malli.core$properties.invokeStatic (core.cljc:1997)
    malli.core$properties.invoke (core.cljc:1994)
#2022-05-0508:15pinkfrogIs there any example that uses malli to validate html forms on the frontend side?#2022-05-0515:03Alexis SchadHi there, is there a minimal sample project using Malli instrumentation (during dev) in a shadow-cljs webapp? Or is there a recommended configuration for https://malli.dev in cljs? I'm struggling a bit to setup it well. -> thread for more info#2022-05-0515:03Alexis SchadWith a basic configuration I got:#2022-05-0515:03Alexis Schad#2022-05-0515:03Alexis Schadwith thrower i got:#2022-05-0515:03Alexis Schad#2022-05-0515:08dvingothe implementation of that is under active development - if you can use the latest git commit you'll have better errors#2022-05-0515:09dvingothis thread has some background, I suggest adding this metadata to your entry ns: https://github.com/metosin/malli/issues/695#issuecomment-1116884390#2022-05-0515:12Alexis Schad> you'll have better errors The error seems pretty good rn, it's just a problem of formatting them inside the console/devtools I'm looking for your link to see if it can improve something#2022-05-0515:21Alexis SchadWith the master version of pretty/reporter, there's a reader conditional for cljs and it now prints well, thanks! I'll see if I have hot reload issue like your link#2022-05-0515:30dvingonice! yea it's tricky because instrument outputs code where it is called that overwrites the fn - so if you're iterating on the implementation of the the fn in a separate namespace they can get out of sync#2022-05-0515:43Alexis SchadI was using dev/start! but I had to use instrument! instead. I think I don't need the {:dev/always true} metadata, it seems to just work fine without it (I call instrument! every time the app is hot reloaded).#2022-05-0515:45dvingoYou'll run into staleness issues described in the ticket without reloading the ns where instrument is called#2022-05-0515:46dvingoif you macroexpand the intstrument call it can help understand why that happens#2022-05-0516:07Alexis SchadI tried to update both the function and the spec and it seems to work well#2022-05-0518:15dvingonice! well alright then lol#2022-05-0520:09Alexis SchadI didn't even thank you… so thanks!#2022-05-0600:02dvingofor sure! no prob - thanks for trying it out! the leverage from malli is pretty remarkable, just helping to add to that power 🙂#2022-05-0700:36Alexis SchadI think I've just had some issues that has been solved with {:dev/always true} Reloading the webapp was doing nothing, I had to restart my shadow-cljs watch. And with the workaround above it seems to work.#2022-05-0619:38mauricio.szaboHey, @danvingo, I though maybe if you want, we could use a thread here to discuss this PR? https://github.com/metosin/malli/pull/702. I added some comments on how to make the metadata work with Shadow-CLJS, but it only works on ClojureScript :thinking_face:#2022-05-0622:06dvingogot something working! these macros are fun 😄 thanks for investigating, I thought adding metadata on the symbol would work but I'm guessing the cljs compiler handles functions differently#2022-05-1312:39eskosI’d need a bit of help/ideas to implement conditional validation for a case when • vector with a single Thing validates as true as it normally would • vector with multiple Things validates true only if each Thing has an additional field • Thing is a record represented as map Practically this is “If only one record is present in result set, id field isn’t required, otherwise id field is mandatory”. There’s probably multiple ways to tackle this, I’m just not sure what would be a good approach.#2022-05-1313:22eskosBasically what I have is
(m/schema [:or
           [:vector {:min 1} (mu/merge IdField Thing)]
           [:vector {:max 1} Thing]])
and this seems to work as intended but I’m not sure if this is the best way to do this. EDIT: Ordering also matters, the more complex schema must be first for sensible error output.
#2022-05-1613:55ikitommiWould :multi here?
[:multi {:dispatch count}
 [1 [:vector {:max 1} Thing]]
 [:malli.core/default [:vector {:min 2} (mu/merge IdField Thing)]]]
#2022-05-1614:04eskosOh yes, :multi seems to work wonderfully! 👍#2022-05-1613:55ikitommiWould :multi here?
[:multi {:dispatch count}
 [1 [:vector {:max 1} Thing]]
 [:malli.core/default [:vector {:min 2} (mu/merge IdField Thing)]]]
#2022-05-1405:25PrashantI was trying to write a function schema such that specs of second argument are dependent on the value of first argument e.g. first argument can be a :keyword with value :id or :contact-id second argument has to be a :map that needs to contain the key supplied in first argument So far, I have done below:
[:=>
   [:cat
    [:alt [:= :id] [:= :contact-id]]
    [:alt [:map [:id :string]] [:map [:contact-id :string]]]]
   :string]

(defn test-fn
  {:malli/schema =>test-fn}
  [key value]
  (str (name key) "_" (key value)))
However, above allows (test-fn :contact-id {:id "op"}) as a valid case in instrumentation as there is no correlation b/w first and second argument. I would greatly appreciate some help. EDIT: Below schema works:
(def =>take2
  [:=>
   [:cat
    [:alt
     [:cat [:= :id] [:map [:id :string]]]
     [:cat [:= :contact-id] [:map [:contact-id :string]]]]]
   :string])

(defn test-fn-2
  {:malli/schema =>take2}
  [key value]
  (str (name key) "_" (key value)))
(test-fn :contact-id {:id "op"}) is invalid with instrumentation on. It is a little verbose though.
#2022-05-1611:24vemvI might need a refresher. Given data expressed as strings (because the data comes from something that inherently only expresses things as strings e.g. a spreadsheet or csv) and a richer schema (i.e. attribute x is an int, not a string), how do I coerce those string values into a schema? Example:
(my/coerce [:sequential [:map [:a int?]]]
           [{:a "1"}]) ;; => Produces `[{:a 1}]`, i.e. coerces the string into an int
;; ...and throws if no coercion was possible
#2022-05-1611:26ikitommitry decoding: https://github.com/metosin/malli/blob/master/README.md#value-transformation#2022-05-1611:28ikitommiit does not throw by default, but you can validate the result and throw if it's invalid. Two walks on the schema with malli are still much faster than one (decode or throw) with most others.#2022-05-1611:28vemvthanks much! Let's give it a go :)#2022-05-1613:50chen florescu
(def example-schema
  [:map {:closed true}
   [:product {:optional true} Product]
   [:version {:default "1.0"} [:= "1.0"]]
   [:enable true?]
   [:window [:int {:min 12 :max 1512}]]])
I have this schema and I want to add a constraint that only if I receive product (which is optional), then window needs to be [:int {:min 12 :max 24}]. Any suggestions how can I do that?
#2022-05-1615:34Ben SlessThose are two different schemas and there's an or between them. Then product won't be option, either#2022-05-1709:03eskosCould a new version be released to clojars? I just spent an embarrassing amount of time wondering why (mt/default-value-transformer {::mt/add-optional-keys true}) isn’t working only to realise it was added 18 days after 0.8.4 was released 😛#2022-05-1812:59Ferdinand BeyerIs there a schema for any homogenous collection, similar to spec’s coll-of? I found :vector , :sequential and :set, and fell into the trap that a set is not sequential?. I want to spec a function that will take any collection and passes it to (set).#2022-05-1813:05Ferdinand BeyerWould it be worth adding a :coll schema to base-schemas like this?
:coll (-collection-schema {:type :coll, :pred coll?})
#2022-05-1911:21armedHey, everyone. What’s the best way to describe homogeneous map but add specific keys in it?#2022-05-1911:22armedI tried :union and :and with no luck.#2022-05-1911:23armed
(mu/union
   [:map
    [:state keyword?]
    [:failure {:optional true} any?]
    [:value any?]]
   [:map-of keyword? string?])
#2022-05-1911:29armedWell, actually it’s working. Had to replace :union with mu/union 🙂#2022-05-1913:01ikitommigood to hear it works! I think union puts :or around the two here.#2022-05-1913:02ikitommi:thinking_face:#2022-05-1913:02ikitommibut, there is an issue to solve this elegantly.#2022-05-1922:27aaron51Seems like pretty/explain doesn’t support custom registries?#2022-05-2014:16plinshello everyone, I’m using reitit and malli to validate JSON requests to endpoints of my API I would like to use the malli registry, in a spec fashion, to define keys first, and afterwards referencing them in a map. doing this way would allow me to https://clojure.org/about/spec#_decomplect_mapskeysvalues the problem Im facing is: the json payload will spawn un-qualified keys, and the registry only works with qualified keys what would be the best solution to this problem? I would like to avoid [:key :ns/key]#2022-05-2306:07jprudentHello, I've a question about the behaviour of :and. Here is a schema with a valid input:
(m/explain [:and
              [:sequential any?]
              [:multi {:dispatch 'first}
               [:a [:sequential keyword?]]
               [::m/default any?]]]
             [:a :b])
=> nil
#2022-05-2306:09jprudentbut with invalid input
(m/explain [:and
              [:sequential any?]
              [:multi {:dispatch 'first}
               [:a [:sequential keyword?]]
               [::m/default any?]]]
             1)
Execution error (IllegalArgumentException) at malli.core/-multi-schema$reify$reify$fn (core.cljc:1497).
Don't know how to create ISeq from: java.lang.Long
I was expecting that the branch that checks that the input is sequential would prevent execution of the second one
#2022-05-2306:13jprudentThe solution I found to avoid an exception to be raised is to have smarter dispatch function
(m/explain [:multi {:dispatch (fn [x] (if (seq? x) (first x) ::error))}
              [:a [:sequential keyword?]]
              [::error [:sequential any?]]
              [::m/default any?]]
             1)
=>
{:schema ...,
 :value 1,
 :errors ({:path [:muguet.meta-schemas/error],
           :in [],
           :schema [:sequential any?],
           :value 1,

           :type :malli.core/invalid-type})}
#2022-05-2306:23jprudentWhy checking branches in an :and is not lazy (could stop after one of the branch is invalid) ?#2022-05-2306:24jprudent(I can live with current behaviour, no problem 🙂 just curious) Thanks#2022-05-2310:06Alexis SchadHi. I'm very not expert in malli, but maybe it's the use of explain: explain try to give all the reasons why it's not valid. Let's say you have a password that must have a specific length, some special characters, etc. You may want to display all the reasons at once instead of a "one by one" strategy. But a lazy version would be useful I think. (I didn't check but validators should already be lazy)#2022-05-2312:30jprudentYou're right, validate is "lazy"!#2022-05-2312:31jprudentYour explanation makes sense, thanks.#2022-05-2311:48rovanionIs it possible to remove a key-value-pair from a map during decode if the value is a specific thing? I've got a select with an entry that reads "all" which is functionally the same as not giving any value at all and I want to handle that case in the reitit/malli request param coercion stage.#2022-05-2311:52rovanionI think I've found a relevant example:
(m/decode
  [string? {:decode/string {:enter 'str/upper-case}}]
  "kerran" mt/string-transformer)
#2022-05-2311:52rovanion
; => "KERRAN"
#2022-05-2311:56rovanionThough that won't remove the key:
(malli.core/decode [:map [:a [:string {:decode/string #(if (= "all" %) nil %)}]]]
                   {:a "all"}
                   malli.transform/string-transformer)
;; => {:a nil}
#2022-05-2312:44rovanionPara on IRC gave me a solution:
(defn- remove-alla-odlingsplatser
  "Takes a query-params map and removes the key :odlingsplatser if its value is 'alla odlingsplatser'."
  [m]
  (if (= "alla odlingsplatser" (:odlingsplats m))
    (dissoc m :odlingsplats) 
    m))   
#2022-05-2312:44rovanionWelll, the solution he gave me was: (fn [m] (if (= "all" (m :a)) (dissoc m :a) m))#2022-05-2315:13William RobinsonI want to use malli/decode on a map that has to contain at least one of :foo/id URI or :bar/id UUID. No matter how I twist and turn the schema, the only thing I can get working is the following:
(malli/decode [:or
                 [:map
                  [:foo/id uri?]
                  other-child
                  other-other-child]]
                 [:map
                  [:bar/id uuid?]
                  other-child
                  other-other-child]]]
    {:bar/id    #uuid "15a0bc27-5725-41d9-89c9-6f8d3966447a"
     :other-key :other-value}
    (mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
Is there a good way to handle this without having to duplicate the schema?
#2022-05-2315:57jprudent
[:map
               [:foo/id [:or [uri?] [uuid?]]]
               [:attr1 :int]
               [:attr2 :int]]
#2022-05-2315:57jprudentcould that work ?#2022-05-2316:28William RobinsonThanks for the suggestion! Unfortunately they are different keys, one of which always corresponds to a URI and the other to a UUID.#2022-05-2316:32William Robinson
[:map
               [:or [:foo/id uri?]]
                    [:bar/id uuid?]]
               [:attr1 :int]
               [:attr2 :int]]
Something like the above is what I would like to achieve.
#2022-05-2317:05jprudent
[:union
 [:or
         [:map [:foo/id uri?]]
         [:map [:bar/id uuid?]]]
 [:map
  [:attr1 :int]
  [:attr2 :int]]] 
#2022-05-2317:07William RobinsonThis looks promising, will try it out and let you know, thanks!#2022-05-2317:08jprudent
[:and [:map
       [:foo/id {:optional true} uri?]
       [:bar/id {:optional true} uuid?]
       [:attr1 :int]
       [:attr2 :int]]
 [:fn (fn [m] (or (:foo/id m) (:bar/id m)))]] 
#2022-05-2408:49William RobinsonForgot to get back to you, but the last solution worked like a charm, huge thanks!#2022-05-2406:40Ben SlessI'm trying to have mutual recursion with regex schemas and failing pitifully. Any tips?#2022-05-2406:47Ben SlessIn this example:
(defn char-range
  [from+to]
  (let [from (long (first from+to))
        to (long (second from+to))]
    (m/-simple-schema
     {:type ::char-range
      :pred (fn [x] (<= from (long x) to))})))

(defn char-seq
  [cs]
  (into [:cat] (mapv (fn [c] [:= c]) cs)))

(m/parse
 (m/schema
  [:schema
   {:registry
    {::digit (char-range "09")
     ::lower-case (char-range "az")
     ::upper-case (char-range "AZ")
     ::letter [:alt ::lower-case ::upper-case]
     ::number [:+ ::digit]
     ::var [:cat ::upper-case [:* ::lower-case]]
     ::boolean [:altn [:true (char-seq "true")] [:false (char-seq "false")]]
     ::atom [:cat ::lower-case [:* ::letter]]
     ::term [:altn
             [:equality [:schema [:ref ::equality]]]
             [:atom ::atom]
             [:var ::var]
             [:number ::number]
             [:boolean ::boolean]
             #_[:structure ::structure]
             #_[:list ::list]
             #_[:string ::string]]
     ::equality [:cat ::var (char-seq " = ") ::term]
     }}
   ::equality])
 (seq "X = Y"))
::equality succeeds but ::term doesn't
#2022-05-2407:31jprudentI'm not sure why but I also find out that you can't have :ref in seqexp. Is that your problem ?#2022-05-2407:33jprudentIs it going in a stackoverflow if you remove it ?#2022-05-2407:34jprudentmaybe try :orn instead of :altn#2022-05-2409:02Ben SlessThat doesn't work#2022-05-2409:04Ben Slessfor some reason you also can't wrap the ::term in ::equality with [:schema [:ref ,,]]#2022-05-2418:31Alexis SchadWhat doesn't succeed? What do you expect?#2022-05-2418:31Alexis SchadI got [[\X []] [\space \= \space] [:var [\Y []]]]#2022-05-2419:14Ben SlessTry replacing ::equality schema with ::term#2022-05-2420:07Alexis Schadho ok#2022-05-2420:08Alexis SchadI think it's because in ::term you altn only the \space and not the whole input seq. It tries to match \space and don't find any entry. I'll try to add an entry to check that#2022-05-2420:18Alexis Schad
(m/parse
 (m/schema
  [:schema
   {:registry
    {::digit (char-range "09")
     ::lower-case (char-range "az")
     ::upper-case (char-range "AZ")
     ::letter [:alt ::lower-case ::upper-case]
     ::number [:+ ::digit]
     ::var [:cat ::upper-case [:* ::lower-case]]
     ::boolean [:altn [:true (char-seq "true")] [:false (char-seq "false")]]
     ::atom [:cat ::lower-case [:* ::letter]]
     ::term [:* [:altn
                 [:equality [:schema [:ref ::equality]]]
                 [:atom ::atom]
                 [:var ::var]
                 [:number ::number]
                 [:boolean ::boolean]
                 [:space [:cat [:= \space]]]
                 [:equals [:cat [:= \=]]]
                 #_[:structure ::structure]
                 #_[:list ::list]
                 #_[:string ::string]]]
     ::equality [:cat ::var (char-seq " = ") ::term]}}
   ::term])
 (seq "X = Y"))
This "works" => [[:var [\X []]] [:space [\space]] [:equals [\=]] [:space [\space]] [:var [\Y []]]]
#2022-05-2502:45Ben SlessWell, you changed the semantics of the parser. An "equals" is defined in the bnf as "atom = term"#2022-05-2502:45Ben SlessI don't want to do extra parsing afterwards#2022-05-2502:46Ben Sless”X = Y” should be tagged as an equality term#2022-05-2507:18Alexis SchadI know, that's why I put " around works. I was trying to explain the reason why it doesn't work. Btw you can’t do recursive seqexp in malli actually. If you want to exclusively use Malli to parse, you can replace the ref to ::term with a [:* any?] and call the parser again on that. You can also create your own schema but it’s harder I think.#2022-05-2508:39Ben SlessWhy can't I have recursive seqex schemas, though?#2022-05-2508:44Alexis SchadDon't know, not a malli expert. But I think it's due to technical limitation. Malli explicitely prevent it with ::potentially-recursive-seqex error, though it is theorically possible.#2022-05-2508:51Alexis SchadSee https://github.com/metosin/malli/blob/85a948afc13262adb2082ede36ee131bd2b6364d/test/malli/core_test.cljc#L1797-L1800 [:schema {:registry {::ints [:cat int? [:ref ::ints]] is disallowed.#2022-05-2705:28escherizeThere’s more info here in this discussion: https://github.com/metosin/malli/pull/317#2022-05-2516:47kennyHi. Instead of an :error/fn returning a string, could I return explain data? If yes, what's the recipe to do so?#2022-05-2516:58Alexis SchadWhat do you mean by "explain data"? But yes if you returns a non-string value, it works as you expect (I think).#2022-05-2612:45plinshttps://clojuredocs.org/clojure.spec.alpha/explain-data#2022-05-2915:14ikitommiCurrently no, but in one project needed richer explanations and did an ugly hack to enable that. I think we should support that at library level. E.g. one could return {:message "so wrong", :level "warning"}#2022-05-2915:15ikitommiCould you write an issue of this?#2022-05-2911:09ingesolIs it possible to make the code below work on the JVM, or is there some limitation in the implementation of CLJS function schemas that only makes them accessible in the JS runtime? My function schemas work and reload nicely in the browser, but in the code below the collect-cljs call always returns () . Additional info: All my function schemas are :malli/schema metadata.
(defn -main
  [& _]
  (-> (malli.clj-kondo/collect-cljs) 
      (malli.clj-kondo/linter-config) 
      (malli.clj-kondo/save!)))
#2022-05-2915:12ikitommiGood question. I have no idea! I believe @danvingo knows.#2022-05-2921:21dvingoif you have a symbol in your functions schema like:
(defn minus
  {:malli/schema [:=> [:cat :int] [small-int]] }
  [x] (dec x))
small-int can only be resolved at runtime by a js vm. that's the fundamental limitation to why it won't work on a java vm
#2022-05-3004:57ingesolThanks for clarifying, @danvingo. Also, big thanks for all your recent work on the CLJS tooling, it’s made a huge difference!#2022-05-3015:31dvingono problem. Ah that's great to hear! glad it's working for you - interested to hear if you have any feedback or things that could be improved, but if all is well that's great too 🙂#2022-05-3015:57dvingoin case you didn't know there's https://github.com/metosin/malli/blob/384e31b325928a29f6bfdddae3a9f1241be4734b/src/malli/clj_kondo.cljc#L203 which is intended for use in a cljs repl - it's manual, you have to copy and paste to the kondo config. Are you trying to output the kondo config using a script (somewhat automated)? One thought I had is that for scripting support we could add a write function for node.js, the only annoying thing is that any browser APIs not available in node.js would probably break things, which would mean you'd have to change the structure of your code to put the function with schemas in one place. There's tradeoffs in all these approaches unfortunately#2022-05-3016:16ingesolYeah, that sounds a bit too much, I was thinking about node as well but probably not the route to go. My use case: I would like to keep the clj-kondo config in version control. The ideal would be a process that recompiles the config on every code change, so it would be obvious for the developer that the config is changed and needs to be pushed.#2022-05-3016:18ingesolbeing able to print it to console and then copy into the file is not too bad, will probably be fine.#2022-05-3016:20ingesolAs for the general usability of CLJS: I’m using shadow-cljs. I followed the advice in the docs, and things seem to mostly work. I think I stumbled upon a file here and there that wasn’t being updated when schema changed, but it usually is fixed by a refresh.#2022-05-3016:21ingesolDon’t have much detail on that here, as I’m not actively working on that code right now. But I will in a few weeks, will try to investigate properly then. But current state of things is perfectly fine for me!#2022-05-3113:29dvingoI see, the use-case makes a lot of sense, I'm wondering if there is a way to achieve it as part of the build process in CI. The tricky part is you want the code to be evaluated in a browser, but you also want access to the filesystem. I was thinking about something like karma tests - running in a headless browser in node... Not sure if that would work.#2022-06-0115:13dvingoAnother idea to get this a bit more automated: add a dev-time only button to your app that sends the cljs kondo config to a specific endpoint which writes to the kondo config file - so it's just a button press#2022-06-0115:14ingesolYeah, that could work. Might be our preferred solution in the end 🙂#2022-06-0115:19dvingooh - or just do it in your reload hook! then no interaction needed#2022-06-0205:51ingesolTrue, that’s what I’m doing right now, but of course just printing to console.#2022-06-0205:53ingesolQuestion: I’m exclusively using keywords in schemas, never symbols. At least for now. Would I be able to “cheat” by telling the malli->cljs-kondo API that my cljs codebase is clj?#2022-06-0215:08dvingoyou could put them in cljc files and they'd be picked up when you collect clj schemas. Other than that, take a look at the source, the code is pretty tiny that collects the schemas, so you should be able to get it working (I think) with your own kondo wrapper. The library can't make those assumptions about what user schemas will have in them, but with those assumptions I don't think there's a reason it can't work#2022-05-3004:57ingesolThanks for clarifying, @danvingo. Also, big thanks for all your recent work on the CLJS tooling, it’s made a huge difference!#2022-05-3004:52AbhinavWhen malli transforms a data structure, does it not preserve meta data? is this by design?
(require '[malli.core :as mc]
         '[malli.transform :as mt])

(def ^{:a 10} alist (list 1 2 3))

(meta (mc/decode [:vector int?] alist (mt/transformer (mt/json-transformer) (mt/string-transformer))) ) 
;=> nil
#2022-05-3111:58eskosThis might be a bit broad question, but tl;dr is How do I implement a custom transformer? I think I want to implement a completely custom transformer to take over the entire decode/encode process - that is, I want to entirely bypass what the built-ins do, and bring my own logic to do the value transformation. I have something brewing elsewhere that I want to integrate to malli which does value type transformation on its own, and for it to work I essentially need a function callback on malli side which would get the relevant malli schema node and its properties (eg. ) + the input value from data to be able to do that transformation. To me this looks like a job for custom transformer which would be created with the malli.transform/transformer helper function, but I can’t make heads or tails about what kind of code I need to write to actually enable such overriding capture of the entire transformation.#2022-05-3112:20ikitommitry :default-decoder with :compile if you want to prepare the transformer to be fast#2022-05-3112:24ikitommiHere’s a sample, if that’s what you are looking for:
(def transformer
  (mt/transformer
   {:name :my-transformer
    :default-decoder {:compile (fn [schema _]
                                 (println "<<<" (pr-str schema))
                                 (fn [x]
                                   (println ">>>" (pr-str schema) "->" (pr-str x))
                                   x))}}))

(def decode
  (m/decoder
   [:map
    [:x [:set [:enum "S" "L"]]]
    [:y :uuid]
    [:z [:tuple :boolean [:map [:a :int]]]]]
   transformer))
;<<< [:map [:x [:set [:enum "S" "L"]]] [:y :uuid] [:z [:tuple :boolean [:map [:a :int]]]]]
;<<< [:malli.core/val [:set [:enum "S" "L"]]]
;<<< [:set [:enum "S" "L"]]
;<<< [:enum "S" "L"]
;<<< [:malli.core/val :uuid]
;<<< :uuid
;<<< [:malli.core/val [:tuple :boolean [:map [:a :int]]]]
;<<< [:tuple :boolean [:map [:a :int]]]
;<<< :boolean
;<<< [:map [:a :int]]
;<<< [:malli.core/val :int]
;<<< :int

(decode
 {:x #{"S" "L" "XL"}
  :y :invalid
  :z [true {:a 123}]})
;>>> [:map [:x [:set [:enum "S" "L"]]] [:y :uuid] [:z [:tuple :boolean [:map [:a :int]]]]] -> {:x #{"L" "S" "XL"}, :y :invalid, :z [true {:a 123}]}
;>>> [:malli.core/val [:set [:enum "S" "L"]]] -> #{"L" "S" "XL"}
;>>> [:set [:enum "S" "L"]] -> #{"L" "S" "XL"}
;>>> [:enum "S" "L"] -> "L"
;>>> [:enum "S" "L"] -> "S"
;>>> [:enum "S" "L"] -> "XL"
;>>> [:malli.core/val :uuid] -> :invalid
;>>> :uuid -> :invalid
;>>> [:malli.core/val [:tuple :boolean [:map [:a :int]]]] -> [true {:a 123}]
;>>> [:tuple :boolean [:map [:a :int]]] -> [true {:a 123}]
;>>> :boolean -> true
;>>> [:map [:a :int]] -> {:a 123}
;>>> [:malli.core/val :int] -> 123
;>>> :int -> 123
#2022-05-3112:31ikitommiIf you want to run bottom-up, use :leave:
(def transformer
  (mt/transformer
   {:name :my-transformer
    :default-decoder {:compile (fn [schema _]
                                 (println "<<<" (pr-str schema))
                                 {:leave (fn [x]
                                           (println ">>>" (pr-str schema) "->" (pr-str x))
                                           x)})}}))
#2022-05-3112:31ikitommi=>#2022-05-3112:31ikitommi
(decode
 {:x #{"S" "L" "XL"}
  :y :invalid
  :z [true {:a 123}]})
;>>> [:enum "S" "L"] -> "L"
;>>> [:enum "S" "L"] -> "S"
;>>> [:enum "S" "L"] -> "XL"
;>>> [:set [:enum "S" "L"]] -> #{"L" "S" "XL"}
;>>> [:malli.core/val [:set [:enum "S" "L"]]] -> #{"L" "S" "XL"}
;>>> :uuid -> :invalid
;>>> [:malli.core/val :uuid] -> :invalid
;>>> :boolean -> true
;>>> :int -> 123
;>>> [:malli.core/val :int] -> 123
;>>> [:map [:a :int]] -> {:a 123}
;>>> [:tuple :boolean [:map [:a :int]]] -> [true {:a 123}]
;>>> [:malli.core/val [:tuple :boolean [:map [:a :int]]]] -> [true {:a 123}]
;>>> [:map [:x [:set [:enum "S" "L"]]] [:y :uuid] [:z [:tuple :boolean [:map [:a :int]]]]] -> {:x #{"L" "S" "XL"}, :y :invalid, :z [true {:a 123}]}
#2022-05-3112:50eskosAlright, thank you! What’s the second argument in :compile (fn [schema _] ... ?#2022-05-3112:56ikitommi_options, passed in all callbacks. Escape hatch to override anything at any time 😉#2022-05-3112:57eskos*danger-zone* 😄#2022-06-0517:08eskosContinuing on this, I’d need a bit of clarification, see snippet:
(malli.core/decode
  [:map [:a {:foo :bar} :int]]
  {:a "string"}
  (malli.transform/transformer
    {:name :hello
     :default-decoder
     {:compile (fn [schema _]
                 (println (str "node schema: " (malli.core/-form schema)))
                 (fn [x]
                   (println (str "value:" x " / schema at val: " (malli.core/-form schema)))
                   x))}}))
node schema: [:map [:a {:foo :bar} :int]]
node schema: [:malli.core/val {:foo :bar} :int]
node schema: :int
value:{:a "string"} / schema at val: [:map [:a {:foo :bar} :int]]
value:string / schema at val: [:malli.core/val {:foo :bar} :int]
value:string / schema at val: :int
=> {:a "string"}
It seems the :a tuple is walked twice, once for the tuple itself and once for the tuple’s value. I find this confusing, as the associative nature of [:a :int] would imply the properties of the key apply to the value as well. There’s a pretty easy way to get around this, schema
[:map [:a [:int {:foo :bar}]]]
does provide properties for the :int which really was my intention, and it sort of makes sense, but it’s of course a bit kludgier. My assumption here was that if, for example, the key is marked optional then its value is naturally optional as well, and as such, all other properties should transfer to values directly as well, plus the value shouldn’t be walked on its own. So, bug, feature, gotcha or something I could trick the transformer to handle through the aforementioned _options so that it doesn’t walk to the tuple’s value? 🙂 EDIT: I created an issue about this, lets continue on GH
#2022-06-0612:27Tiago Dall'OcaHello hello! Is there a generics implemented in malli in some way? I was thinking maybe e.g. [:generic=> [:n-type number?] [:cat :n-type] [:vector :n-type]], :generic=> taking a vector of pairs of keywords and schemas and then the usual :=> args#2022-06-0613:50Noah BogartAll clojure functions are generic, lol#2022-06-0615:05Tiago Dall'Ocathe idea is to parse malli schemas to ts type definitions#2022-06-0615:05Tiago Dall'Ocato make cljs libraries nicer to consume from ts#2022-06-0615:12Noah BogartSorry, I replied with a cheeky comment but I meant to follow up: clojure inherently doesn't have generics and while I don't want to speak for the metosin folks, i suspect that the concept of "generics" fundamentally doesn't align with schema systems like malli. you can already achieve generics by just using :or and normal clojure functions: [:or (mapv #(do [:=> [:cat %] [:vector %]]) [number? keyword? string?])]#2022-06-0615:13Noah Bogarthaving said that, iwould be interested to see what you could build in this space as i think it has potential. it's cool to see the overlap between clojure schema systems and typescript's structural typing#2022-06-0616:10Ben SlessSimple schema can take an argument, make it the type#2022-06-0616:12Ben Slesshttps://github.com/metosin/malli#content-dependent-simple-schema#2022-06-0712:23Tiago Dall'Ocavery nice#2022-06-0712:23Tiago Dall'Ocamalli positively surprising me as usual#2022-06-0712:23Tiago Dall'Ocathank you all for the excellent work!#2022-06-0612:28Tiago Dall'OcaI'm developing mali-ts as a way to integrate malli schemas into typescript's type system and it'd be really good to have generics, though I understand why in clojure it might not make so much sense#2022-06-0612:29Tiago Dall'OcaMaybe it's not even that hard to implement? I'm not sure#2022-06-0612:31Tiago Dall'OcaAnd on the topic typescript integration, it'd be really good to have a way to model Promises and async stuff
#2022-06-0616:13Ben SlessIf you have generics modeled then promises are solves, no?#2022-06-0612:31Tiago Dall'OcaIf doesn't make sense to have those in malli, I might implement in malli-ts only#2022-06-1219:55borkdudeHi! Can this be fixed somehow?
------ WARNING #1 -  -----------------------------------------------------------
 File: ~/.m2/repository/metosin/malli/0.8.4/malli-0.8.4.jar!/malli/core.cljc:2265:26
--------------------------------------------------------------------------------
2262 |          (reduce -register-var {}))))
2263 | 
2264 | (defn class-schemas []
2265 |   {#?(:clj Pattern, :cljs js/RegExp) (-re-schema true)})
--------------------------------^-----------------------------------------------
 References to the global RegExp object prevents optimization of regular expressions
I'm compiling malli as part of #nbb and I'd like to keep the rest of nbb optimized
#2022-06-1220:18borkdudeFYI: https://github.com/babashka/nbb#metosinmalli#2022-06-1220:19juhoteperihttps://github.com/metosin/malli/pull/692 Needs new release#2022-06-1220:36borkdudeAwesome thank. cc @ikitommi#2022-06-1304:47ikitommiawesome! I’ll try to cut a release today (lot’s of small changes waiting)#2022-06-1308:08ingesolI tested function instrumentation in CLJS recently, and it works pretty well with the pretty printer. One thing I noticed as a pain point was that I wanted to see the input args when an output value does not match the schema. Would that make sense to add? Or did I simply miss it?#2022-06-1309:01ikitommiit’s not passed to the error reporter, should be a small change: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2453#2022-06-1309:02ikitommiissue and pr would be most welcome.#2022-06-1309:02ikitommino, the args are passed in#2022-06-1309:03ikitommijust not used: https://github.com/metosin/malli/blob/master/src/malli/dev/pretty.cljc#L51-L58#2022-06-1321:10ingesolThanks. Was a quick fix, posted a PR#2022-06-1605:01pinkfrogHi, is there some tool like https://github.com/Provisdom/spectomic but generates datomic schema from malli instead of spec ?#2022-06-1605:39Ben SlessI tried writing one once, I guess you could port some ideas from here and write a transformer like there is for json schema#2022-06-1607:37pinkfrogAre you saying porting ideas from the above repo?#2022-06-1607:48eskosI don’t think Malli’s transformers can do 1-to-n replacements (eg. single value to map), although I haven’t tested the behavior for such case. That said, I don’t think it should be that hard to create equivalent behavior with some helper functions and whatnot, plus one area of malli I haven’t explored is the raw walk api, which may work just fine for this 🙂#2022-06-1608:54juhoteperiShould work. E.g.: [:map {:decode/string (fn [s] (if (string? s) (js/JSON.parse s) s))} [:foo :keyword]] Decode function for the map schema is run before checking map properties, and decoding those. At least I've used this in cases where API takes comma separated string, and it is decoded to seq/vec and then items are coerced to keywords or such. Not sure if this answers the original question, but 1-to-n transfomer tranformations should work.#2022-06-1608:55juhoteperi(decoding is one use/type of transformer)#2022-06-1611:49eskosOkay, that’s great. Tangentially, I just did a Thing™ which I didn’t mean to but can be abused to do something like this 😅 I’ll create another thread about it since it’s sort of a pre-release now.#2022-06-1611:53eskosThere’s that post. So yeah, https://github.com/esuomi/muotti/ can be abused like so:
(require '[malli.core :as malli])
(require '[muotti.core :as muotti])
(require '[muotti.malli :as mm])

(malli/decode
  [:string]
  1
  (mm/transformer (muotti/->transformer 
                    {:transformations {[:int :string] {:transformer (constantly {:a 1 :b "wow"})}}})))
Obviously custom schema registry support needs to be added to make this even remotely sensible :rolling_on_the_floor_laughing:
#2022-06-1611:51eskosJust released https://github.com/esuomi/muotti/, a graph-based value transformation chain library which has integration with malli’s transformers.#2022-06-1611:52eskosMalli support is still somewhat in progress as malli has tons of schemas and some can be considered quite subjective (eg. for value transformation, what does ) but maybe those can be supported eventually as well. For now core types are supported.#2022-06-1611:54eskosAnd this is a sort of pre-release for now which is why I put this here, not #announcements - have to let the dust settle first 🙂#2022-06-1612:19ikitommiLooks great! like byte-streams, but for (literal) data. Malli will have effective and derived types, will simplify things a lot, https://github.com/metosin/malli/issues/264#2022-06-1612:26eskosThanks! 😸 Figuring out which schemas to support and how is going to be an on-going process, I think - I’m going to exclusively avoid doing validation with muotti, naturally. My intention is to start using this in an actual project quite soonish, so I’m hoping that’ll let me refine the native support and providing sensible defaults. And as curiosity, muotti was made almost entirely during a few train trips between Helsinki and Tampere… 🙂#2022-06-1801:35ValentínHi, I want to use malli for form-input validation... I want to display customs errors#2022-06-1801:36ValentínHow could I display error msg base on the input value?#2022-06-1801:39ValentínExample: If the user did not type anything at all, I want to display "This field is required", is the user typed just one character, I want to display the following error msg "It should be at least 2 character long"#2022-06-1801:41Valentín
(def form-name [:string {:min 2
                         :fn {:error/fn '(fn [{:keys [value]} _]
                                           (if (empty value)
                                             "This field is required"
                                             "It must be at least 2 character long"))}}])
(def form [:map [:name form-name]])

(me/humanize (m/explain form {:name ""}))
; The behaviour I want
; => "This field is required"

(me/humanize (m/explain form {:name "J"}))
; The behaviour I want
; => "It must be at least 2 character long"
#2022-06-1808:34George PeristerakisI think your error is in your precondition, the function you should use is empty? instead of empty#2022-06-1808:43Ben SlessI think the defaults already give you this behavior. Try deleting the fn property#2022-06-1814:58ValentínThere is also something wrong, the fn function never evaluate. @UK0810AQ2 I want to custom the default error msg.#2022-06-1815:42Ben SlessThe property should be :error/fn directly, without the preceding :fn key But what's wrong with the default errors?
:string {:error/fn {:en (fn [{:keys [schema value]} _]
                             (let [{:keys [min max]} (m/properties schema)]
                               (cond
                                 (not (string? value)) "should be a string"
                                 (and min (= min max)) (str "should be " min " characters")
                                 (and min max) (str "should be between " min " and " max " characters")
                                 min (str "should be at least " min " characters")
                                 max (str "should be at most " max " characters"))))}}
And if it's in a map you'll get "missing required field blah" from there Also, I think you should use empty? and not empty
#2022-06-2003:55Valentín@UK0810AQ2 did you run this code on the repl? I coud not make it works.#2022-06-2003:58ValentínAlso, I copy and paste the following code from github, and I'm getting the following exception... What am I missing?#2022-06-2003:58ValentínThanks#2022-06-2003:59Ben SlessI copied this example from malli's source, I assume it works#2022-06-2420:44jprudent@U0103KEKSLR you need to add the sci dependency to your project if your functions are quoted#2022-06-1801:42ValentínThanks in advance#2022-06-2004:06PanelCould it be done to validate with an async fn ?#2022-06-2016:52ikitomminot sure what is the question in here.#2022-06-2012:27borkdudeHow do you express [s/Long] (vector of Long in Schema) in malli?#2022-06-2012:32ikitommia [:vector :int] is close, but for all int?s.#2022-06-2012:32ikitommifor exact class, many ways (could be simpler), but one being:
(def a-long (m/-simple-schema {:pred #(instance? Long %)}))

(m/validate a-long (Integer. 12)) ;=> false
(m/validate a-long (Long. 12)) ; => true
#2022-06-2012:33ikitommiwith min, max & json-schema translation:
(def a-long
  (m/-simple-schema
   {:type :long
    :pred #(instance? Long %)
    :json-schema {:type "long"}
    :property-pred (m/-min-max-pred nil)}))

(m/validate [a-long {:min 10}] (Long. 12)) ; => true
(m/validate [a-long {:min 10}] (Long. 8)) ; => false
#2022-06-2012:34ikitommisomeone asked full support for using Java Classes as Schemas. But that would not work with cljs.#2022-06-2012:34ikitommiwould allow [:vector Long]#2022-06-2012:35borkdude@U055NJ5CC The reason I'm asking is that my new (work-in-progress) CLI library is using a notation similar to schema for coercion: https://github.com/babashka/cli E.g.: {:coerce {:a :int}} will turn "--a" "1" into {:a 1} and {:coerce {:a [:int]}} will turn "--a" "1" into {:a [1]} . This coercion syntax doesn't have to be as powerful as malli. The idea is that you can coerce command line args into a map and then do validation on that (in your clojure function) using spec or malli (since it's a detail whether you are calling this function from the command line or from the REPL)#2022-06-2012:39ikitommiyeah, that looks great!#2022-06-2012:40ikitommithere is the malli-lite syntax too.#2022-06-2012:40ikitommi
(defn coercer [schema]
  (m/decoder (lite/schema schema) (mt/string-transformer)))

((coercer {:a :int}) {:a "123"})
; => {:a 123}
#2022-06-2012:47ikitommiso, you will have your own transforming part there (the simplest way to do this, looks simple), but the future processing would be done using a full(er) schema/spec lib?#2022-06-2012:48borkdudeThe idea is inspired by clojure -X, you just call a clojure function from the command line basically. And whether you call this function via the command line, or via the REPL, the validation of arguments needs to happen anyway. So why put the validation logic in a CLI library, while it should probably live inside your app code#2022-06-2012:49borkdudeAnd so yes, you should do validation of args the same way you were doing it anyways in Clojure (with malli, schema, manual asserts)#2022-06-2012:49borkdudeAnd you can then choose whatever you were using. The main job of the CLI library is to transform strings into data, not much more than that#2022-06-2012:52borkdudeBy keeping it simple, you can also put your coercion "spec" in the deps.edn file (it doesn't require any function symbols, just keywords and collections)#2022-06-2012:55ikitommi👍#2022-06-2012:56ikitommithat’s one of the goal of malli too, to give a literal notation for schemas. Using full malli is too big for this case I guess? or does it miss something?#2022-06-2012:57borkdudeI want to it be an un-opiniated library so it works together with malli, spec, schema or your hand-rolled things#2022-06-2012:58borkdudealthough we could add some docs on how you can integrate with malli#2022-06-2012:58borkdudethe malli-lite syntax seems great for that#2022-06-2012:58ikitommiI think that’s a good choice. Is it possible to make it pluggable?#2022-06-2012:58ikitommiyes, happy to provide the example/glue if there is an extension point for that.#2022-06-2012:59borkdudewe could maybe do this using a protocol or multimethod? :thinking_face:#2022-06-2013:00borkdudebut even without this, it's easy to plug in your own thing, since after coercion you're dealing with just clojure data#2022-06-2114:42ikitommiwithout yet looking at the code, multimethod sounds good. Global Side Effects For The Win! 🙂 will check the repo later. thanks for the lib(s), again.#2022-06-2023:44PanelIf I have a form and want to validate on things that can only be done on server. Could I validate a schema with an async call ?#2022-06-2101:43Lucy WangWhy not? Simply send the raw form data to the server using ajax, the server validates it, then send back the result.#2022-06-2103:04PanelI was wondering if this can be describe as a malli schema, maybe using a Fn schemas.#2022-06-2209:03Setzer22Is there a default schema type for dates? Something like :date or :inst (except I tried those but they don't work)#2022-06-2214:56eskosinst? predicate is supported, but not much else probably due to cross-platform support being tricky#2022-06-2210:31ingesolhttps://clojurians.slack.com/archives/CHY97NXE2/p1655893717482109#2022-06-2310:40ikitommifinally had time to release it, the pre-summer-holiday-edition 🌴#2022-06-2321:36devnany chance at compojure-api + malli? is that work that just needs doing or is there a fundamental reason why malli couldn’t be added as an option on compojure-api?#2022-06-2418:13devnlooking now, i suppose the answer is to use reitit#2022-06-2418:13devnbut either way i’m curious if a patch would be accepted to handle malli coercions in compojure-api#2022-06-2716:19Ben Sless@valtteri moving discussion of malli here, regarding json schema, the first item we should tackle, and hardest, IMO, is time schemas. There was a PR I opened and a long thread on it in the issues. It seems like the biggest tension there is good-enough vs. specification compliance How do you want to handle it?#2022-06-2716:22Ben Slesshttps://github.com/metosin/malli/issues/501#2022-06-2716:22Ben Slesshttps://github.com/metosin/malli/issues/49#2022-06-2716:27dvingofor cljs I would vote for https://github.com/henryw374/cljc.java-time (what tick uses) but how do you deal with this now being a dependency of malli?#2022-06-2716:27dvingoperhaps utilizing https://github.com/borkdude/dynaload ?#2022-06-2716:33valtteriHmmm personally I’d like to minimize dependencies#2022-06-2716:34valtteriI think the java.time stuff looks good and in CLJS we could perhaps just use js dates (I know, they’re bad) or goog.date?#2022-06-2716:35valtteriWe can always make initial release under malli.experimental.something so we can move on and get feedback#2022-06-2716:36dvingohow about a code path branch using dynaload? - one set of schemas backed by js dates and one by js-joda (cljc.java-time)#2022-06-2716:36valtteriAnd if it’s a decision between “good enough” and “specification compliance” this one leans to “good enough”#2022-06-2716:37dvingobut yea not a big deal because a date lib can always be added in user-space#2022-06-2716:39valtteriWould these be realistic goals? • good and practical defaults for Java • “good enough” defaults for JS • no new deps With a quick glance Ben’s suggestion for the java impl looks good and practical to me. I’ll promise to read through all the linked references tonight.#2022-06-2717:01Ben SlessThanks I'd also try getting feedback from Henry, both on our current options and maybe on a path forwards, how we could reach full spec compliance while delivering something that covers 99% of use cases in the meanwhile#2022-06-2717:20valtteriOh, so you have a hard requirement for the full spec compliance?#2022-06-2717:22valtteriI think we anyway want to have extension points so that the behaviour can be customized.#2022-06-2717:23valtteriBut have defaults that work for the 99%#2022-06-2718:44Ben SlessWith an eye towards an end goal of using json schema or async api across an organization, with a variety of languages, I want to at least have parity in features#2022-06-2813:59Colin P. HillUnsure if this is in the scope of your discussion, but I'm currently working on an effort that uses the support Malli has for this so far, and besides a lack of built-in support for types beyond date-time, the other pain point I've been running into for time types is that the generated JSON schemas are less restrictive than the Malli schemas they're based on. A uuid? in Malli becomes a value with a string with a format of date-time in JSON – but JSON Schema validators are not required to treat formats as normative. This means another system could consume the JSON Schema output to validate a message, pass the message to the system using Malli, and then have it rejected as invalid. My workaround has been manually adding a pattern field.#2022-06-2917:08valtteriI've been thinking this and one way to circumvent the date/time type problem could be to treat them as strings. There could be a regex schema according to the format in JSON-schema. Dunno if this would make any sense though. But this is how it works in JSON-schema.#2022-06-2917:10valtteriI've read through all the earlier conversations and checked a couple of different JSON schema implementations how they do it. I think I'm more confused now than in the beginning. 😄#2022-06-2917:13Ben SlessWe could always ask in their slack#2022-06-2917:14valtteriYeah. But what do you think @UK0810AQ2 if in JSON-schema temporal types are strings with some enforced format.. would it make sense to do it in the same way on malli side?#2022-06-2917:15valtteriSo we could be compliant and move on 😉#2022-06-2917:15valtteriAnother thing we should clarify is that which version(s) of the JSON-schema spec are we targeting to support#2022-06-2917:26valtteriWe can also try to not solve the malli temporal schemas in general but make json-schema specific variants. If this would help us to move on#2022-06-2919:18Ben SlessThing is, in malli you can accept strings and make sure they transform to timestamps. When they leave the system they'll be serialized back to strings anyway#2022-06-2919:19Ben SlessThe temporal thing has become my bugbear#2022-06-2919:34valtteriYeah I understand that but I was thinking since timestamps are the “blocker” because there are so many possible ways to do them and it's difficult to make it good for both java and js, we could try to avoid those decisions. Java side is more clear to me. We could also think “java first and js some day”.#2022-06-2919:36valtteriIf we think only Java and assume your suggestion about the types, there are no blockers…right? :thinking_face:#2022-06-2919:44Ben SlessNot from me, Tommi or Henry might have reservations we should account for#2022-06-2919:50valtteriI think we could whip up a working example to make it concrete. I guess the types can easily be changed if better suggestions pop up#2022-06-2919:51valtteriI mean a working example in context of json schema#2022-06-2919:52valtteriIt's easier to get that feedback when people can see it in action#2022-06-3018:13valtteriOh man, I just noticed that the old json-schema->malli PR is actually alive again!#2022-06-3018:18valtteriI don’t understand how I had missed that. I’ll comment to the PR that we’re thinking about these same things here. Dunno if https://github.com/tangrammer is in this Slack?#2022-06-2800:35DerekCross-posting from #announcements : Announcing org.passen/malapropism. A small library for configuration data backed by malli https://github.com/dpassen/malapropism#2022-06-2803:04pinkfroghttps://www.metosin.fi/blog/high-performance-schemas-in-clojurescript-with-malli-1-2/ Do we have a second series?#2022-06-2804:50Alexander MoskvichevIs there an easy way to change default date format in malli.transofm? I've read several discussions about custom registry, etc, but just started with malli, it's too hard for now to understand. I use malli with reitit, just need to change outgoing date format#2022-06-2817:47Setzer22Does malli support custom function predicates? I noticed some functions seem to be supported but most of them give an "invalid schema" error and I don't know if there's another way of doing this#2022-06-2817:48Setzer22actually, what I need is define a schema for a map that contains a core.async channel. I can always use :any but it would've been nice to use something like #(instance? ManyToManyChannel %)#2022-06-2817:54Setzer22Also, unrelated question 😄 I'm pretty curious about this: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#tldr In particular, in this snippet:
(defn plus1
  "Adds one to the number"
  {:malli/schema [:=> [:cat :int] :int]}
  [x] (inc x))
What's this way of using metadata to define the schema of the function via defn? Is this a defn-like macro defined somewhere?
#2022-06-2817:58skynet@jsanchezf if I understand, it's the normal Clojure defn and that's the attr-map? argument which becomes metadata https://clojuredocs.org/clojure.core/defn#2022-06-2817:59Setzer22@craigy But then, how does it work? Malli has no way of knowing when a function is redefined. Simply defining some metadata does not lead to instrumentation happening#2022-06-2818:00dvingo
(dev/start! {:report (pretty/reporter)})
check the source of that call for the answer 🙂
#2022-06-2818:01dvingothis section https://github.com/metosin/malli/blob/master/docs/function-schemas.md#defining-function-schemas explains it#2022-06-2818:01dvingo> Without instrumentation turned on, there is no schema enforcement: #2022-06-2818:01dvingo> defn schemas can be defined with standard Var metadata. It allows defn schema documentation and instrumentation without dependencies to malli itself from the functions. It's just data. #2022-06-2818:04Setzer22I see, so looking at the source (https://github.com/metosin/malli/blob/b745b73a93109a71643ce58302189a6f69c56d5e/src/malli/dev.clj#L15) it walks all the namespaces at the time where you call dev/start! , then looks for instrumented functions. But how will it know when I create a new function or define a new namespace? Do I need to call start! again for it to pick it up?#2022-06-2907:25ikitommiyes, you need to call start! again when you annotate new functions. Tried to hook Var-watching for already defined schematized defns, but the Clojure core doesn’t support that easily. Calling start! makes the collecting explicit.#2022-06-2821:42ingesolI’m using malli to instrument functions and print results using the pretty printer. Some functions accept and return huge maps, so huge that it makes the output of the pretty printer unusable. Any hints on how to improve this?#2022-06-2906:27eskoshttps://github.com/greglook/puget can make pretty prints even more prettier; sometimes just adding color helps a lot • https://github.com/lambdaisland/deep-diff2 (or clojure.data/diff) can be used to diff the content; with a bit of trickery you can print only the interesting parts#2022-06-2907:21ikitommi@U8SFC8HLP Puget in nice, but Malli uses a custom pretty printer directly on top of fipp - different opinions about colors & works with CLJS too. About the huge maps, currently there is no omitting of valid values (https://github.com/bhb/expound#show-valid-values), would have needed that too, but have had no time to implement. PR would be welcome on this. Also, I think it’s the final piece before extracting the pretty printing from reitit & malli into a clean and minimal lib (https://github.com/metosin/virhe).#2022-06-2907:25ingesolI think showing valid values is useful to get the full picture. It also makes the report easier to parse sometimes. If there are 4 args to the fn, and 1 is wrong, it can be easier on the eyes to navigate to the wrong one when you use the valid ones as guides.#2022-06-2907:27ikitommiok, I’m all ears on how to make it better then.#2022-06-2907:27ingesolIn my case, I have a re-frame app-db. The number of keys in the map is not huge, but the number of nested keys is. So some way of sniffing out the approximate size of the map and deciding to print the first 500 chars or something would be nice. That was what I was looking for in my question. Would be nice if fipp had that option.#2022-06-2907:28ingesolFrom reading the code, it looks like we just pass the map to fipp for printing?#2022-06-2907:29ikitommihttps://github.com/metosin/malli/blob/master/src/malli/dev/virhe.cljc#L34#2022-06-2907:29ingesolgroup hug#2022-06-2907:30ingesolbut virhe isn’t currently used in malli, is it? I suppose maybe I can use it to implement an alternative to the default pretty printer?#2022-06-2907:30ikitommivirhe is… inlined 🙂
(ns malli.dev.virhe
  "initial code for "
#2022-06-2907:30ingesolso… what I’m asking for is already available?#2022-06-2907:31ikitommiactual virhe-repo is just a README, “copy code here when it’s ok”#2022-06-2907:31ikitommiyes#2022-06-2907:31ikitommiyou can either swap the EDNPrinter or pass options to it to work differently#2022-06-2907:31ikitommiall code is in malli repo.#2022-06-2907:34ingesolawesome, thanks! I just didn’t notice that part of the API. Will have a go at this 🙂#2022-06-2909:26ingesolIn case it’s useful for anyone, here’s the change I needed. I wanted to avoid printing the re-frame app-db. Both because the contents are usually not interesting, and because it’s way too large to be printed effectively. In the case of a value not matching the schema of our app db, it would usually be something that does not match the check here so it would be printed anyway.
(extend-type v/EdnPrinter
  fv/IVisitor
  (visit-map [this x]
    (if (:some-key-always-in-our-db x)
      ;; Avoid printing app db to console, as it is way too large
      (v/-color :text "{... app db ...}" x)
      ;; Inlining original implementation from EdnPrinter
      (let [xs (sort-by identity (fn [a b] (arr/rank (first a) (first b))) x)]
        (fe/pretty-coll this (v/-color :text "{" this) xs [:span (v/-color :text "," this) :line] (v/-color :text "}" this)
                        (fn [printer [k v]]
                          [:span (fv/visit printer k) " " (fv/visit printer v)]))))))
#2022-06-2822:10steveb8nmaybe #portal?#2022-06-3014:43stathissiderishello, does anyone know what the :schema schema does?#2022-06-3015:17ikitommiit's an eager reference, works like indentity. You can for example escape a sequence schema with it: [:cat :int [:* [:schema [:cat :int]]]#2022-06-3015:18stathissideristhank you! I thought you were on vacations 😉#2022-06-3018:06borkdudeA PR which bridges the gap between malli and #babashka! https://github.com/metosin/malli/pull/718#2022-07-0109:41Eric DvorsakIs there a Malli equivalent to clojure.spec/every? I'm trying to write this spec in Malli https://github.com/clojure/core.specs.alpha/blob/master/src/main/clojure/clojure/core/specs/alpha.clj#L39#2022-07-0109:57valtteriHmmm I'm not sure if that exists directly but I guess it can be implemented with :and ?#2022-07-0112:20Eric Dvorsakyeah that's what I'm trying but can't really figure out how to achieve the same thing. every allows to name and declare specs for different key/values in the map as tuples#2022-07-0111:16borkdude#2022-07-0403:41kokonutDoes malli have anything equivalent to Union in Typed Racket? https://docs.racket-lang.org/ts-guide/beginning.html
#lang typed/racket
(define-type Tree (U leaf node))
(struct leaf ([val : Number]))
(struct node ([left : Tree] [right : Tree]))
It is the same concept with F#'s discriminated union. I read malli documentation carefully but couldn't find.
#2022-07-0404:29Ben SlessIsn't this just or?#2022-07-0408:00hansbuggeOr :multi which has support for discriminators via :dispatch https://github.com/metosin/malli#multi-schemas#2022-07-0408:51Ben SlessIs the racket Union discriminating?#2022-07-0410:09kokonut@UK0810AQ2 I believe you can say that.#2022-07-0410:15Ben Slessmulti can dispatch to the correct schema which would make it discriminating. If you don't have a way of doing that, you can use or or orn, which is a named union#2022-07-0409:39Eric DvorsakI'm still trying to spec clojure.core with malli https://github.com/clojure/core.specs.alpha/blob/master/src/main/clojure/clojure/core/specs/alpha.clj#L39 I'm trying to find a way to replace this every which specs a map as 3 different kinds of key/values. unfortunately Malli map-of takes only one kind of key I'm wondering if there is a way to transform the values in the schema, so that it checks that it's a map then transforms in a repetitions of tuples. Is it possible with Malli? going through the docs I can't find any clear way#2022-07-0410:03Eric DvorsakI think I found my solution in malli source https://github.com/metosin/malli/blob/2398df55ee806e25592fabf4d0c642ee3a2b233f/src/malli/destructure.cljc#L19#2022-07-0410:19Eric DvorsakIs it expected that the output of m/parse can be lossy?
(defn -map-like? [x] (or (map? x) (and (seqable? x) (every? (fn [e] (and (vector? e) (= 2 (count e)))) x))))
(def MapLike (m/-collection-schema {:type 'MapLike, :empty {}, :pred -map-like?}))
(m/parse [MapLike [:orn
                                [:some-keyword [:tuple :keyword :any]]
                                [:some-int [:tuple :int :any]]]]
                                   {:k1 1
                                    :k2 2
                                    3 4
                                    5 :six})
{:some-keyword [:k2 2], :some-int [5 :six]}
In this snippet above one can see that because the output of parse is a map, keys of the same tuple are overwritten
#2022-07-0500:22Stig BrautasetHow to model a deck of cards? We use spec at work, but I wanted to try Malli for a personal toy project. I promptly ran into a problem I couldn't solve, and I've tried to make a simplified version of it here. Imagine a simplified game of poker: a hand is 5 cards, and we only care about whether it is a flush (i.e. 5 cards of the same suit) or not.#2022-07-0500:23Stig BrautasetFirst attempt As we don't really care about the numbers or faces we can model a card as simple enum of its suit, and a hand as a 5-tuple of cards. The deck is trickier. It has up to 52 cards, but I can't find a way to model the additional constraint that we should only have a maximum of 13 of each suit. Thus the last test in this self-contained example (usually) fails:
(ns cards.simple-test
  (:require
   [clojure.test :refer [deftest is testing]]
   [malli.core :as m]
   [malli.generator :as mg]))

(def Card [:enum :spades :hearts :clubs :diamonds])
(def Hand [:tuple Card Card Card Card Card])
(def Deck [:sequential {:max 52} Card])

(deftest deck
  (testing "we can generate a Deck"
    (doseq [deck (mg/sample Deck)]
      (testing "that validates"
        (is (true? (m/validate Deck deck))))

      (testing "where every element is a card"
        (is (every? #(m/validate Card %) deck)))

      (testing "of maximum 52 cards"
        (is (>= 52 (count deck))))

      (testing "where each suite occurs at most 13 times"
        ;; this test (usually) fails, because we haven't restricted
        ;; the count of each suit to maximum 13.
        (is (>= 13 (->> deck
            frequencies
            vals
            (reduce max 0))))))))
#2022-07-0500:24Stig BrautasetSecond attempt Let's give each card a suit and number to address the problem with our previous deck. A hand is now slighly more complicated: we have to use a set with {:min 5 :max 5} properties instead of a 5-tuple to ensure cards are not duplicated. (A hand with 5 ace of spades should not be valid!) The deck, on the other hand, is simpler than before. By using a set we can drop the {:max 52} property. It is now implicit, because there are only 52 possible combinations of 4 suits and 13 numbers. Self-contained example:
(ns cards.better-test
  (:require
   [clojure.test :refer [deftest is testing]]
   [malli.core :as m]
   [malli.generator :as mg]))

(def Card [:tuple
	   [:enum :spades :hearts :clubs :diamonds]
	   [:int {:min 1 :max 13}]])
(def Hand [:set {:min 5 :max 5} Card])
(def Deck [:set Card])

(deftest deck
  (testing "we can generate a Deck"
    (doseq [deck (mg/sample Deck)]
      (testing "that validates"
        (is (true? (m/validate Deck deck))))

      (testing "where every element is a card"
        (is (every? #(m/validate Card %) deck)))

      (testing "of maximum 52 cards"
        (is (>= 52 (count deck))))

      (testing "where each suite occurs at most 13 times"
        (is (>= 13 (->> deck
            (map first)
            frequencies
            vals
            (reduce max 0))))))))
#2022-07-0500:25Stig BrautasetThis solves the problem with my first attempt, but introduces another: we can neither sort nor properly shuffle our deck now. 😞#2022-07-0500:34Stig BrautasetI think what I want is an ordered sequence of distinct values, but that may be because I can't see outside my Specs-shaped box. Is there an alternative way to model this with Malli?
(def Deck [:sequential {:distinct true} Card])
#2022-07-0516:26roklenarcicDefine card as tuple of suit and number up to 13. A deck is a distinct sequence of cards. Unshuffled deck is a sorted deck#2022-07-0516:37Stig BrautasetThat sounds like my second attempt:
(def Card [:tuple
	   [:enum :spades :hearts :clubs :diamonds]
	   [:int {:min 1 :max 13}]])
(def Hand [:set {:min 5 :max 5} Card])
(def Deck [:set Card])
#2022-07-0516:38Stig BrautasetHow can I specify a distinct sequence? It seems you have to go to a set, as there’s no (documented) {:distinct true} property I can add.#2022-07-0518:57Stig BrautasetAh, I think I see. I just discovered :fn 😄#2022-07-0519:01Stig BrautasetThis appears to do what I want:
(def Card [:tuple
	   [:enum :spades :hearts :clubs :diamonds]
	   [:int {:min 1 :max 13}]])

(def Hand [:and
           [:sequential {:min 5 :max 5} Card]
           [:fn (fn [hand] (apply distinct? hand))]])

(def Deck [:and
           [:sequential Card]
           [:fn (fn [deck] (apply distinct? deck))]])
#2022-07-0519:05Stig BrautasetThat can be made to work with my simplified version too:
(def Card [:enum :spades :hearts :clubs :diamonds])
(def Hand [:sequential {:min 5 :max 5} Card])
(def Deck [:and 
           [:sequential Card]
           [:fn #(->> % frequencies vals (reduce max 0) (>= 13))]])
Thank you for the help 🙂
#2022-07-0519:06Stig Brautasetduckie strikes again 🙂#2022-07-0501:59dumrathttp://malli.io not working?#2022-07-0717:58valtteriWorks for me :thinking_face:#2022-07-0616:39raymcdermottdoes anyone have experience / examples of using malli to generate openAPI docs that https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-create-api-from-example.html?#2022-07-0717:46valtteriIs there something special with ApiGW or should normal swagger 2.x work? There is malli.swagger namespace..#2022-07-0717:46valtteriOpenApi 3.x has been requested and it’s somewhere on the backlog but not under active development atm.#2022-07-0718:07raymcdermottI was thinking that the swagger ns would probably be OK but would be nice if somebody has already trodden the path. That's all really.#2022-07-0718:21valtteriPlease let me know how it turns out. 🙂#2022-07-0717:25bortexzHi, does malli support specifying the relationship between arguments and return on function schemas, similar to spec’s fdef :fn ? Couldn’t find anything related when going over the Readme#2022-07-0717:57valtteriHmmm I guess we don’t have that. :thinking_face: Feel free to file an issue on github. Also contributions are welcome!#2022-07-0809:56bortexzFound an existing one, will keep an eye on it 🙂 https://github.com/metosin/malli/issues/608#2022-07-0923:45PanelWould anyone has anything to share regarding generating Web Form from Malli ? I have seen multiple conversations, blog post and even conf talk but none of those have code. Thanks#2022-07-1007:52eskosI suppose it would require a bit indirect and more involved route to approach, since having somesuch library would need to make assumptions about nesting, specific form events, usability issues (think tab navigation, grouping, aria tags...). So maybe the reason is that no one's just yet done a good enough system they're comfortable with abstracting to a library 🙂#2022-07-1008:08valtteriThis is something we're doing ad-hoc in different projects. We've discussed about wrapping the best practices into a lib and open sourcing it. But I'd say the status is still “cooking”#2022-07-1008:45PanelSounds like the interceptor pattern, everyone using it code a new implementation to fit their needs. @U6N4HSMFW can you share describe how you do it ? Do you walk the schema and generate hiccup ? Do you have 2 data structures, one for the validation and one to describe UI specifics ? One option I played with is to use a JS lib that accept json-schema, that did not work out for you ?#2022-07-1010:30valtteriJSON-schema is fine for simple cases and if that works for you then that’s a perfectly valid way to go in my opinion.#2022-07-1010:31valtteriWe’ve tried it both ways you described. Just a couple of months ago a colleague demoed “walking malli schema and generating hiccup” approach.#2022-07-1010:31valtteriI think the “generate ui from schema” approach pays off if you have really complex forms and a lot of them#2022-07-1010:32valtteriOr when you need to dynamically generate the forms#2022-07-1010:33valtteriI think you could also get quite far with something like Formik + malli validation#2022-07-1700:08geraldodev@U6N4HSMFW the malli piece is set on stone for me, as for the react library, could you please elaborate why that were choosen ? Was it the better at the time, has it a feature that more inline with the approach that you are using. I've started to look into react-hook-form https://github.com/geraldodev/react-form-hook-test/blob/use-form-state/src/main/app/core.cljs#L45 but I've stopped. Going back to it soon. Today my colleague pointed me to https://github.com/tannerlinsley/react-form , In the last two weeks I'm learning that code from tanner have high quality, I briefly looked today one example. It looked that it gets too deep into validation. It would be nice a library that interact with DOM (react) just enough and gives as the hooks (not react) points to bring malli awesomeness to our beloved clojure data strutures. I'm following this rabbit hole. Are you too ?#2022-07-1110:22Stig BrautasetSay I want a schema for an integer in the range 1-10 (inclusive). Is there a practical difference between these two? If so, which one is recommended?
repl> (mg/sample [:int {:min 1 :max 10}])
(2 1 3 1 7 4 7 4 10 3)

repl> (mg/sample [:and :int [:>= 1] [:<= 10]])
(1 1 1 2 5 1 9 6 6 2)
The former is shorter, and it's more intuitive to me to start with :int when that's the base type. But I don't know if there's a benefit to using :>= and :<= to enforce the bounds instead.
#2022-07-1111:40valtteriI would use the one that reads better: :min :max#2022-07-1111:43valtteriAt least I'm not aware of any extra benefits. :>= and :<= compile a bit differently compared to :min :max https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2281-L2290 vs. https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L589-L598 and if I had to guess, I'd say the :min :max might be a little bit faster. But this is handwaving. 😉 And the difference is probably very small.#2022-07-1112:21Stig BrautasetThank you!#2022-07-1314:36George PeristerakisI have a registry with large data objects specs. I created a list of base specs to have a more consistent taxonomy ex: :pk-int [int? {:min 1 :unique true}] . What is the best way to represent these base specs? Should I have a separate registry or keep them in the same registry?#2022-07-1316:23DrLjótssonNewbie question: How do I specify that a value should be of a specific class, for example java.time.ZonedDateTime#2022-07-1316:51respatialized[:fn #(instance? % java.time.ZonedDateTime)]#2022-07-1316:52respatializedNot sure if there's another "official" way but I've done it that way in the past#2022-07-1316:58DrLjótssonThanks! I thought there might be a more official way but this will do just fine. #2022-07-1319:32Tiago Dall'OcaHey, more of a implementation question: why schemas instances are implemented with reify instead of using records? When I tried extending IPrintWithWriter to malli.core.Schema it didn't work because the type generated for schemas with reify isn't malli.core.Schemas#2022-07-1404:59LoicI have a question regarding key-transform and :map-of : I have map keys that are list and I want to turn that list into vector of vector. But I struggle to make a basic transformation work such as just turning keys into string in case of use of :map-of
;;work

(m/decode
 [:map [:smth :int]]
 {:a 2}
 (mt/key-transformer {:decode name}))
=> {"a" 2}

;; does not work
(m/decode
 [:map-of any? int?]
 {:a 2}
 (mt/key-transformer {:decode name}))
=> {:a 2}
#2022-07-1511:04Ferdinand BeyerIs there a built-in way in Malli to strip away map keys that are not in the schema? Let’s say I have an open map schema like this:
[:map [:person/first-name string?]
      [:person/last-name string?]]
And a compliant value with an extra key:
{:person/first-name "Cosmo"
 :person/last-name "Kramer"
 :person/likes #{:fruit}}
I’d like to “sanitise” this data and only keep specified keys, recursively.
#2022-07-1511:08Ferdinand BeyerThanks for acting as a sound stage. I found a simple way that can easily extended for recursive application:
(require '[malli.util :as mu])

(select-keys data (mu/keys schema))
#2022-07-1511:16valtteriThere's a built-in strip-extra-keys-transformer for value transformation https://github.com/metosin/malli#value-transformation#2022-07-1511:19Ferdinand BeyerAmazing, thanks! Missed that one 😊#2022-07-1511:19valtteriNo problem 🙂#2022-07-1913:15AJ JaroWe recently updated a functions schema definition and it wasn’t written correctly, but not caught during the tests. Is there a way that Malli can be configured to validate arguments during Clojure tests? It was caught during “manual” testing but not during the automated tests#2022-07-1918:52respatializedIn your test namespace:
(require '[malli.instrument :as mi]
[clojure.test :as t])

(defn instrument-fixture [f]
(mi/collect!)
(mi/instrument!)
(f)
(mi/uninstrument!)
)

(t/use-fixtures :once instrument-fixture)
#2022-07-2320:13hanDerPederThis works
(def ok-schema
  {:thing/create [:map
                  [:status [:string {:default "a"}]]]})
I want to extract the status part into a reusable schema, so I do
(def bad-schema
  {:thing/status :string
   :thing/create [:map
                  [:status [:thing/status {:default "a"}]]]})
This is not valid. Throws an exception:
No implementation of method: :-into-schema of protocol:
   #'malli.core/IntoSchema found for class: clojure.lang.Keyword
Hopefully it’s clear what I’m trying to do. Is this a bug or am I going about this the wrong way?
#2022-07-2414:55pppaul[:string}#2022-07-2520:57hanDerPederhttps://github.com/metosin/malli/pull/729#2022-07-2414:58pppauli'm trying to edit malli.core, i can't figure out how to step through (get into a working cider debugging state) -map-schema. i tried pulling out the validctor method into it's own fn and setting that to cider-debug-defun-at-point but without success. any tips?#2022-07-2415:01pppaul
(m/schema
    [:map {:closed true}
     [:optional {:optional true} [:enum :a :b]]
     [:specified :int]])
code i'm using to try to trigger -map-schema
#2022-07-2416:08pppaulI've also tried this, and i get a stack overflow error. i'm not really sure how to approach developing for malli.
(m/validate
  (m/schema
    [:schema {:registry {:map (-map-schema)}}
     [:map {:closed true}
      [:optional {:optional true} [:enum :a :b]]
      [:specified :int]]
     ])
  {
   ;;:hi        "valid"
   :specified 1
   ;;"bad" :error
   }
  )
#2022-07-2416:09pppaulthis is where i have imported -map-schema into my ns and can eval it... i did this because of the stack overflows i'm getting when debugging. however, when i don't go into debug mode i don't get problems, when i'm in debug mode i get stack overflows pretty wild#2022-07-2606:53Yehonathan SharvitI am surprised by how JSON transformer works
(m/decode :int "123" malli.transform/json-transformer) ;; => "123"
Why isn’t the string "123" converted to an integer? In the same way,
(m/decode [:map [:foo :int]] {:foo "123"} malli.transform/json-transformer) ;; => {:foo "123"}
#2022-07-2608:23vemvmalli.transform/-json-decoders doesn't have a string->double thingy maybe you want to be looking at malli.transform/string-transformer instead?#2022-07-2609:24Yehonathan SharvitThe distinction between json-decoders and string-decoders is not 100% clear to me#2022-07-2609:29vemvMy impression is that string-transformer is for parsing string-based values, and json-transformer for enriching values from already-parsed json with JVM/clj types (uuid, date, keyword etc) not present in json So it might make sense to compose string-transformer with json-transformer#2022-07-2610:42Yehonathan SharvitIt makes sense to me also#2022-07-2610:55Yehonathan Sharvit@U45T93RA6 Is there a transformer that keywordize map keys when the schema keys are keywords?#2022-07-2610:58vemvlet's move it 1:1#2022-07-2613:54pppaulI have a list of keyseqs [[:parent :child1 :child2]...]that map in a human readable way into my schema, but when i do mu/get-in mu/update-in the mappings don't work. when i do subschemas the paths are sorta like [:parent 0 0 :child1 0 0 :child2] and :parent will have 10 different paths to the same :child1 and :child2. it seems a bit like a flip of a coin if i get the correct path or not. i've tried a few different ways of figuring out the correct one (smallest count). also the :in and :path are different, and i'm not sure what to use in mu/update-in. unfortunately i'm not in control of the schema (it's generated based off of json-schema). my next approach will involve recursive mu/find-in, but i'm wondering if anyone has run into a problem like this before, where conceptual indexing and real malli indexing have a significant mismatch and how to get to the correct spots in the schema to programmatically edit it (i'm adding custom properties)#2022-07-2705:44Martynas MIs there a way to represent this as a map?
{id: string?
 size1: my-schema1
 size2: my-schema1
 size3: my-schema1}
Where I know that id will be present all the time and size1 and others are dynamic. i.e. they can exist or they can have completely different names. I could use map-of for the second part but I would like to use map for the id part.
#2022-07-2708:11Yehonathan SharvitInteresting question @U028ART884X#2022-07-2708:13Martynas MInstead I hardcoded the keys. It was simpler than think and solve for truly any key 😄#2022-07-2710:37pppaulyes, i posted an issue about something like this on malli's github#2022-07-2710:39pppaul
(let [{:keys [additionalProperties]} form
                                          req-fields                     (->> required (mapv keyword) (into #{}))
                                          all-fields                     (concat props req-fields)]
                                      (if-not additionalProperties
                                        (make-map props req-fields)
                                        [:and
                                         [:fn (fn [m]
                                                (->>
                                                  (apply dissoc m all-fields)
                                                  (m/validate (m/schema [:map-of 'string? 'string?]))))]
                                         (make-map props req-fields)]))
#2022-07-2710:39pppaulthat's code i use to transform a json schema map to act something like what you described#2022-07-2710:40pppaul
[:and
 [:fn (fn [m]
        (->>
          (dissoc m :optional :required)
          (m/validate (m/schema [:map-of keyword? string?]))))]
 [:map
  [:required [:enum :a :b]]
  [:optional {:optional true} [:enum :a :b]]
  ]]
ends up making something like that
#2022-07-2710:42pppaul
make-map (fn [props req-fields]
                   (apply vector :map
                     (->> props
                       (mapv (fn [[k v]]
                               (if (get req-fields k)
                                 [k v]
                                 [k {:optional true} v])))
                       (sort-by first))))
#2022-07-2710:45pppaulthere is some discussing in malli about making :closed be a schema, supporting this idea with the work around that i posted, but it's not got a PR or anything, so work around is best. you can replace the :map-ofwith a schema for each key very easily with the code i provided#2022-07-2819:36nonrecursivehi all! i’m using reitit with malli for parameter coercion, and I have a schema with default values for some parameters, but those default values aren’t being added. has anyone else run into this?#2022-07-2819:37nonrecursivemy router looks like this:
(def router
  (rr/router routes
             {:data {:coercion   reitit.coercion.malli/coercion
                     :middleware route-middleware}}))
#2022-07-2822:00pppaulthere is a default values transformer#2022-07-2822:13nonrecursiveI discovered the reason, it was because it had {:optional? true} in the options#2022-07-2820:54Noah Bogartis it possible to say "this is either a set or the specific keyword :bypass "?#2022-07-2820:55Noah Bogartrealized my mistake immediately, lol. the answer is: [:or set? [:enum :bypass]]#2022-07-2823:48Mateo DifrancoWhat would be the best way of handling multiple possibilities of a map inside a schema, based on a top level keyword? Maybe I can explain myself better with some examples, let's say I have something like this:
{:something "test"
 :type :a
 :deep {:depends 3}}
and
{:something "test"
 :type :b
 :deep {:another [1 2]}}
So, based on :type, I'm expecting different things in :deep. This seems to be a good case for :multi, given it allows using a function for dispatch. But the dispatch function would take the :deep map, not the top level whole map. I have a work-around for this, but it's quite ugly and I'm wondering if there's a better way.
#2022-07-2902:02pppaulthis looks like what multi is designed for, i haven't used multi, i'm curious where it fails in this case.#2022-07-2914:32Mateo DifrancoWell, I might be wrong, but take a look at this:
(def Something
  [:map
   [:something string?]
   [:type [:enum :a :b]]
   [:deep
    [:multi {:dispatch :type}
     [:a [:map [:depends int?]]]
     [:b [:map [:another [:vector int?]]]]]]])

(m/validate
 Something
 {:something "test"
 :type :a
 :deep {:depends 3}}) ;=> false
And if I do a really ugly side effect, we can look at what exactly the dispatch function takes:
(def Something
  [:map
   [:something string?]
   [:type [:enum :a :b]]
   [:deep
    [:multi {:dispatch (fn [x] (or (println x) :type))} <-------------
     [:a [:map [:depends int?]]]
     [:b [:map [:another [:vector int?]]]]]]])

(m/validate
 Something
 {:something "test"
 :type :a
 :deep {:depends 3}}) ;=> false, but also prints {:depends 3}
#2022-07-2916:32valtteriYou probably should do the multi for the parent of :deep#2022-07-2916:33valtteriIf you need to dispatch on the :type of the parent#2022-07-2916:36Mateo DifrancoI thought about that, but then how would I avoid duplication of the structure? For example, if I move :multi to the top of the schema, I would need to make each possibility contain the whole structure right? In this case, :a would have a schema matching :something for example. Maybe I can merge in each possibility though, but I was wondering if there was a better solution.#2022-07-2916:48valtteriYep, I guess that's whats you gotta do. You can splice the sub-schemas into smaller chunks for reuse#2022-07-2916:49valtteriThere might be some better way I'm not aware of 🙂#2022-07-2916:17annarcanaare there any SQL libraries that work with malli?#2022-07-2916:35valtteriDepends on what you mean with that. :) We have experimental tool that derives malli schemas from sql table schema#2022-07-2918:03colliderwriteror this (no personal experience): https://github.com/kwrooijen/gungnir#2022-07-2918:05annarcanaYes! This looks like what I had in mind (data-driven SQL schemas)#2022-07-2918:06annarcanaWe are already using malli to generate reitit routes from data models, so one thought was to also generate SQL table schemas from same#2022-07-2918:07annarcanaI suppose any data-driven library like gungnir could work but something that already can do malli->SQL type translation was something I was hoping would be out there somewhere#2022-07-2918:07colliderwriterafaict, this does not do schemas but it seems to have all the parts#2022-07-2918:10annarcanait can do table create/drop migrations which should be enough for my purposes, but I'll have to dig into it#2022-07-2918:12colliderwriteri'd be interested to hear your thoughts#2022-07-3014:36AlejandroHello. Is there a blog post, a documentation page, anything that discusses motivation behind malli? I'd like to know how spec, plumatic/schema, and malli relate to each other.#2022-07-3015:09valtteriHere’s a video where Tommi explains also the motivation https://www.youtube.com/watch?v=MR83MhWQ61E#2022-07-3015:10valtteriComparison and reasoning wrt other tools (schema, spec) starts around 3:30#2022-07-3015:35Alejandrothank you#2022-07-3014:37pinkfrogm/validate returns true/false, what’s the function that throws exception when data is malformed?#2022-07-3014:40pinkfrogThe purpose of doing this is, I want to add code like
(m/validate :int foobar)
inside a function, and when foobar is not int, I want an exception is thrown so I can know some thing bad happens in sentry. Another good to have is, the m/validate (equivalent) can be turned on in dev and off in prod.
#2022-07-3014:54annarcanaYou'll most likely want some combination of explain/`humanize`. (https://github.com/metosin/malli#error-messages) I don't believe malli has any function that deliberately throws an exception, by design, but you can always use throw with an Exception whose payload is the malli explain output.#2022-07-3014:56pinkfrog@annapawlicka Maybe I shall just do (assert (m/validate ,,,))#2022-07-3017:12Noah BogartIn the simplest case, this is what I would suggest doing.#2022-07-3015:12Ben SlessDo you want this with decode, too?#2022-07-3101:31pinkfrogyes!#2022-08-0315:54Ben Sless
you should build something like:

(defn coercer
  [schema options transformer]
  (let [schema (m/schema schema options)
        validator (m/validator schema options)
        decoder (m/decoder schema options transformer)
        explainer (m/explainer schema options)]
    (fn [x success failure]
      (let [ret (decoder x)]
        (if (validator ret)
          (success ret)
          (failure (explainer ret)))))))
#2022-08-1504:37pinkfrog@UK0810AQ2 Say I do not want to define the validators in advance. So is it meaningful to create the validators on the fly or shall I just directly use the m/validate function? Is there any performance discrepancy regarding the two?#2022-08-1505:11Ben SlessYes, closing over a validator is faster than calling validate every time. In what context are you using this?#2022-08-1505:32pinkfrogAcross the codebase, many functions accepts a map but the data type etc is not clear. I want to annotate the functions.#2022-08-1507:01Ben Slessah, you should have said so, you want this: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schema-metadata#2022-08-1508:22pinkfrogUsing function schemas, there is a tendency to spec on all the arguments, but I’d rather spec on one of them.#2022-08-1508:24Ben SlessWhy not spec the others as any?#2022-08-1508:29pinkfrogThat causes confusion.#2022-08-1508:34Ben SlessI think you don't have to spec all the arguments with this macro, no?#2022-08-0305:02Stel AbregoHey y'all, I'm wondering if it's possible in Malli to create a spec for a map with these qualifications: • There are some specific keyword keys that are required (this is simple in Malli) • There can also be an arbitrary amount of keys that are vectors, but they all must be vectors of keywords and their values must be maps. (not straightforward) • Besides the two possibilities above, the map is closed (ex no string keys for example, also not straightforward) The last two things are the weird requirements. Seems like :map can't really support this. I might need to use [:and [:map ... ] [:fn ...] but even then it might be awkward to satisfy #3.#2022-08-0316:23pppaulusing :fn is how i did exactly this#2022-08-0316:24pppaulhttps://github.com/metosin/malli/issues/682#2022-08-0316:24pppaullast post is the solution, unless you want to give malli a PR and do some hard work#2022-08-0316:26pppaul
[:and
 [:fn (fn [m]
        (->>
          (dissoc m :specified :optional :required)
          (m/validate (m/schema [:map-of keyword? string?]))))]
 [:map
  [:required [:enum :a :b]]
  [:optional {:optional true} [:enum :a :b]]
  [:specified :int]]]
make a function that outputs something like this
#2022-08-0322:56Stel Abrego@U0LAJQLQ1 thank you for the help! I'm going to look at this later.#2022-08-0323:01pppauljust be aware that i didn't provide a general function, you need to create one that respects the keys in your malli map schema.#2022-08-0419:19Stel AbregoA little update, I realized that clojure.spec can solve this problem with actually great error messages so I might rewrite in spec. Might write a blog post about it later too.#2022-08-0315:52Ben SlessWhat's the best way to express in malli "a map must contain at least one of a set of kv"?#2022-08-0316:46Noah Bogartis that :or or :multi?#2022-08-0317:27Ben SlessIt's probably :or, but it's gross to have [:and M1 [:or M2.0 M2.1 M2.2]]#2022-08-0317:29Ben SlessI also think this requires some awareness of implementation details to get coercion right, because when dealing with two keys, you'll want to try to coerce both, i.e. [:or [:merge Ma Mb] Ma Mb]]#2022-08-0317:29Ben SlessThat's even messier and for only two keys#2022-08-0317:29Ben Slessnow do that with three keys, the combinations become unreasonable#2022-08-0317:35Ben SlessThe correct thing to do in these situations is compile an execution plan#2022-08-0317:35Ben SlessSort of like pattern matching libraries do#2022-08-0317:45Noah Bogartyou could also do a function schema for this, make the various keys optional and then perform the check in the :fn#2022-08-0318:06Ben SlessI am morally opposed to function schemas#2022-08-0318:19Noah Bogartany particular reason? seems appropriate for this kind of work#2022-08-0319:51Ben SlessThey're completely opaque, not well defined, not serializable, hard up to impossible to generate#2022-08-0319:52Ben SlessThe semantics of "at least one of this group" are clear, don't need a function. It can (should) be another schema type
#2022-08-0319:58Noah BogartThat makes sense to me. I wonder how hard it is to write a new schema type like this#2022-08-0320:16Ben SlessA bad implementation is easy, an efficient implementation is hard 🙂#2022-08-0320:18Noah BogartIs it? Naively, I would expect it to be an or:
(fn [obj]
  (or (contains? obj :a)
      (contains? obj :b)
      (contains? obj :c)))
or something similar. is there more to it?
#2022-08-0320:18Noah BogartMaybe I'm misunderstanding the actual problem#2022-08-0320:21Ben Slessbecause there's a combinatoric explosion, you need for two keys: [:or [:map [:k1 S1] [:k2 S2]] [:map [:k1 S1]] [:map [:k2 S2]]]#2022-08-0320:22Ben Slessfor three keys you have 7 options (I think, I'm tired)#2022-08-0320:22Noah Bogart"At least one of a set" means as long as one exists, it doesn't matter if others exist, so you can just check each one#2022-08-0320:23Noah Bogartcould compile a list of the ones that do exist and then merge them after the fact too#2022-08-0320:23Ben SlessYes, but now make decode, encode, parse and explain work 🙂#2022-08-0320:24Noah Bogarthah I won't be nerd sniped! but yes, if you don't want to directly implement a bunch of stuff, I think I can see the issue more clearly#2022-08-0320:25Ben SlessThat's why I dislike function schemas 🙂#2022-08-0423:58mauricio.szaboFolks, a simple question about .pretty/explain - is it possible to NOT print the full value being matched? The reason is: I have a HUGE data, and it's actually breaking my terminal limit, so I can't actually see the error 😞#2022-08-0521:36ingesolhttps://clojurians.slack.com/archives/CLDK6MFMK/p1656494811406489?thread_ts=1656452534.761089&amp;cid=CLDK6MFMK#2022-08-1006:57ikitommiwould be easy to add an option to drop that. please write an issue. Also, see https://github.com/metosin/malli/pull/731#2022-08-1006:57ikitommie.g. option (or new default) will mask the irrelevant parts of the value. they can be huge.#2022-08-0500:02pppaulyou can write it out to a file#2022-08-0500:07mauricio.szaboNot really, I'm on React-Native#2022-08-0501:02pppaulyou may have to edit some of the functions in the namespace to customize the output.#2022-08-0518:26Colin P. HillIn Malli function signatures, what does the ? prefix on a parameter mean?#2022-08-0519:08valtteriIt means that by convention the function will tolerate nil and possibly other nonsense for those params.#2022-08-0610:10hjrnunesI need to validate a map-like type with metadata. Actually, values are collections - vector-like types - which also have metadata that needs validation. I've played around with the -map-schema and -collection-schema implementations and I can make them support my custom types easily, but I am unsure how I can express the metadata schema. I could use properties but I wonder if that makes sense. The metadata is as important as the rest. Actually it's where the most variation will be. Is there a better way than simply maintaining two schemas - one for the actual type and another for the metadata map - and running validation for each independently?#2022-08-1006:48ikitommicurrently, there isn’t an extension point for validation, like there is for most other schema applications (transforming, generating, …), so there isn’t an easy way to add a generic, properties-based declaration for meta-data, if there was, could be something like:
[:map {:metadata [:map [:closed :boolean]]} [:x :int]]
#2022-08-1006:49ikitommiif you can provide an example how and where you would like to define the metadata schemas, could suggest ways to build such thing. It’s definetely doable, many ways to do that.#2022-08-1006:50ikitommiPlease write an issue with some sample data?#2022-08-1007:12hjrnunesWell, that's the thing. I'm not really sure how to best handle it. I can easily do it via properties like the snippet you shared, but it just felt odd to have schema in the properties. Maybe I'm just overthinking it.#2022-08-1007:26ikitommicould also be a lookup, e.g. [:map {:domain/type ::object} …], so that the metadata-schemas are in a separate registry and looked out with a property-key. You can the create a helper that takes a schema and returns a metadata validator and use that when needed. keeps the validation of metadata and values as separate. should be easy’ish to do with schema walking.#2022-08-0621:36Eugenhi, is there a guide to learn malli for dummies? I am new to clojure and I kind of struggle to get my head around malli. I would like to use it for data modeling - as an exploratory tool. I would like to define some entities and be able to customize the generated output. (have a list of names, companies, etc). I read through the section and I still feel like I am missing some context. For example, I would like to generate company names like: Acme + some random smallish number ?!: ACME 12 I have defined an ID like
(def id [:string {:min 1, :max 20}]) 
and I would like to limit what characters it should contain (it should have only printable characters for example)
#2022-08-0622:07hanDerPederMalli, as spec, leverages the test.check library for generating values. I would start there. There are some very useful guides linked in the readme of that project. #2022-08-0622:08hanDerPederhttps://www.youtube.com/watch?v=F4VZPxLZUdA#2022-08-0622:08hanDerPederThis one seems relevant#2022-08-0622:17Eugenthanks @U013U475882#2022-08-0622:19EugenI think my end goal is very similar to what @U07FP7QJ0 talked about here https://www.youtube.com/watch?v=ww9yR_rbgQs&amp;ab_channel=LambdaIsland . Thank you @U07FP7QJ0 for making the video ! I would love to have a more in-depth presenatation of that. If you have anything please share. BTW, can you share how the project evolved? It's been two years.#2022-08-0813:46pppaul
(def id [:string {:gen/elements [list of names to use] :min 1, :max 20}]) 
#2022-08-0813:47Eugenthanks for all the help, I am getting my head around it - but it's lots of new stuff to learn (test-check).#2022-08-0813:49EugenI am thinking of writing some generators that will leverage data sources like WikiData to get real world examples. This should be more to generate some lorem ipsum style of data.#2022-08-0813:49pppaulif you aren't experienced with generators there is a bit of a learning curve, but it's not too hard. i find programmatically attaching the generators harder than making them.#2022-08-0813:50Eugenthanks for the feedback. That is a question I have - how to use the generators in the app. Since it's a lot of new stuff I am bound to make mistakes, hence me asking.#2022-08-0813:51pppaulif you have schema's that can generate nested things, you will want to control those generators a lot (any? is a very nasty one).#2022-08-0813:52Eugenif I make it I will be bale to write on CV: generator tamer 🙂#2022-08-0813:57pppaulalso, the gen/ properties don't work together, so if you want to use elments->fmap you have to use gen/gen instead. if you are converting to json, you need to watch out for symbol/string/keyword keys causing dupe key issues#2022-08-0813:58Eugendon't know what that means for now, but I'll try to keep an eye out for it.#2022-08-0813:58Eugenthanks#2022-08-0813:59pppaulmaybe not a normal issue to run into, but it was an issue for me when i was making generators for schemas that i converted from json to malli (i didn't own the schema)#2022-08-0814:01pppaul{'key :a :key :c "key" :d} that in json is a map where all keys are the same, and thus invalid#2022-08-0814:03pppaulanyway, biggest issue with generators is the recursive stuff, so if you have any of that you have to take care of it right away and figure out how to limit the recursions, or your data is going to get extremely large#2022-08-0814:04Eugenthanks#2022-08-0814:04EugenI'll try to stay away from recursion#2022-08-0816:44Noah BogartJust opened an https://github.com/metosin/malli/issues/734 for a subtle implementation detail/maybe bug with :tuple schemas: if given a seq, m/encode silently fails to do any encoding instead of coercing to a vector.#2022-08-0823:46Eugenis there a guide/version of this article with malli ? https://cognitect.com/blog/2017/3/24/3xeif9bxaom78qyzwssgwz1leuorh4#2022-08-1006:30ikitomminot exactly that, would you be interested in writing that?#2022-08-1006:32ikitommishould be easy to map that to malli#2022-08-1008:54Eugensounds interesting, but right now I don't have time for that. getting my bearings and it takes quite some time. I will consider this in a few months when I get more confient I did (re)found specmonstah (the third time) https://github.com/reifyhealth/specmonstah/ . On papaer it provides what I need so I am going to work with that and see if it's a good fit for what I am looking for: generate data for a web app to facilitate demos#2022-08-0923:52bortexzHi, recently started playing with Malli, I am trying to understand why this works:
(def Map1
  (mu/optional-keys
   (m/schema
    [:map ::a ::b]
    {:registry (merge
                (m/default-schemas)
                (mu/schemas)
                {::a string?
                 ::b [:merge
                      [:map [::a [:ref ::a]]]
                      [:map [:y string?]]]})})
   [::b]))
but this fails with enum child error:
(def Map2
  (mu/optional-keys
   [:map {:registry (merge
                     (m/default-schemas)
                     (mu/schemas)
                     {::a string?
                      ::b [:merge
                           [:map [::a [:ref ::a]]]
                           [:map [:y string?]]]})}
    ::a
    ::b]
   [::b]))
=> 
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1, :max nil}
Is it a bug? Am I missing something? Edit: Another question that is slightly related (involves registries too) malli.util derivative transforms like mu/optional-keys obfuscate the refs when used inside registry:
(def Map1 
  [:map {:registry {::b string?
                    ::a (mu/optional-keys
                         [:map [::b [:ref ::b]]])}}
   ::a])
=>
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/invalid-ref {:type :ref, :ref :malli-domain-lab/b}
Is there any workaround for this?
#2022-08-1006:42ikitommiHi. I believe it’s a bug, please write an issue.#2022-08-1006:45ikitommiRobust way to define a schema space for an application is to use the global scope, e.g.:
(mr/set-default-registry!
  ;; linear search
  (mr/composite-registry
    ;; default schemas
    (m/default-schemas)
    ;; utils
    (mu/schemas)))

(mu/optional-keys
 [:map {:registry {::a string?
                   ::b [:merge
                        [:map [::a [:ref ::a]]]
                        [:map [:y string?]]]}}
  ::a
  ::b]
 [::b])
#2022-08-1008:16bortexzDone! https://github.com/metosin/malli/issues/736 Thank you, I’ll rearrange how the registries are defined to make it work#2022-08-1007:02ikitommigetting back to computer this week from the long summer vacation, happy to see there the lively discussions here. if there is something clearly not resolved / answered by someone (🙇 for everyone for helping others!), you can mention me in the thread. will try to read the new new issues & prs this week.#2022-08-1010:50AkizHi, I need to change all keys in the schema from :keyword to :string. Is there an utility for this?#2022-08-1011:54ikitommiwhat do you mean by keys here: map keys in values? or in schemas? something else? example would help#2022-08-1012:26AkizSure. Here's an example. I would like to achieve that I don't have to define the output-schema manually. In this case it would be enough to replace keyword with string and vice versa. But It would be great if I could somehow use a transformer to transform only schemas.
(require '[malli.core :as m])
(require '[malli.transform :as mt])

(def input-schema [:map [:name :string] [:type :keyword]])
(def output-schema [:map [:name :string] [:type :string]])

(defn encode
  {:malli/schema [:=> [:cat input-schema] output-schema]}
  [val]
  (m/encode input-schema val mt/string-transformer))
#2022-08-1012:28ikitommiwould this help: https://github.com/metosin/malli/pull/737#2022-08-1012:29AkizYes, this is exactly what I tried unsuccessfully to achieve via schema-walk. Thanks!#2022-08-1012:30ikitommi👍#2022-08-1012:38ikitommiMerged the error-value helpers into master. In case of validation error, you can get just the values in error and/or mask the valid values with something like , like expound does. Should be useful in pointing out (small set of) errors in huge values. feedback welcome, ping @mauricio.szabo:
(def Address
  [:map {:closed true}
   [:id :string]
   [:tags [:set :keyword]]
   [:numbers [:sequential :int]]
   [:address [:map
              [:street :string]
              [:city :string]
              [:zip :int]
              [:lonlat [:tuple :double :double]]]]])

(def address
  {:id "Lillan"
   :EXTRA "KEY"
   :tags #{:artesan "coffee" :garden "ground"}
   :numbers (list 1 "2" 3 4 "5" 6 7)
   :address {:street "Ahlmanintie 29"
             :zip 33100
             :lonlat [61.4858322, "23.7832851,17"]}})

(-> Address
    (m/explain address)
    (me/error-value {::me/mask-valid-values '...}))
;{:tags #{"coffee" "ground" ...},
; :numbers (... "2" ... ... "5" ... ...),
; :address {:lonlat [... "23.7832851,17"], :street ..., :zip ...},
; :EXTRA "KEY",
; :id ...}
#2022-08-1012:39ikitomminext thing would be to integrate this into pretty explaning. I think it would be good to mask out valid values by default? and the options being: 1. mask valid values (default) 2. show full values 3. don’t show value at all#2022-08-1012:41ikitommithe tests also show how to do the same thing for individual errors. e.g. present errors one-by-one - like expound does.#2022-08-1013:15mauricio.szaboYes, I think masking valid values by default is good - it seems "prettier" this way, and makes for less noise IMHO#2022-08-1014:48ikitommihttps://github.com/metosin/malli/pull/738, feedback and testing most welcome:
(malli.dev.pretty/explain
 [:map
  [:id :int]
  [:tags [:set :keyword]]
  [:address [:map
             [:street :string]
             [:city :string]
             [:zip :int]
             [:lonlat [:tuple :double :double]]]]]
 {:id "123"
  :EXTRA "KEY"
  :tags #{:artesan "coffee" :garden}
  :address {:street "Ahlmanintie 29"
            :city "Tampere"
            :zip 33100
            :lonlat [61.4858322, 23.7832851]}})
=>
#2022-08-1018:07Mateo DifrancoAwesome!#2022-08-1109:34ikitommi^:-- if someone can show an real-life example where the value masking is still bad, please share (privately ok too). I guess if one has huge nested maps, masking each irrelevant key separately might still cause large values. Or a list of 100 entries with the last one in failure would cause 99 s, which might not be best. For maps, all the irrelevant keys could be optionally accumulated into single entry on as both key and value and for sequences, a …99 to show how many valids there are.#2022-08-1119:34dangercoderHi, great work on Malli! I’m trying to figure out how I can use malli without duplicating attributes. Is the “What I want” syntax possible in some way?
;; What I have
[:map
 [::order-id :int]
 [::order-multiplier :int]
 [::order-amount decimal?]]

;; What I want
[:map
 [::order-id]
 [::order-multiplier]
 [::order-amount]]

(m/validate ::order-id "123")
;; => false
#2022-08-1119:41valtteriYou can define a registry and register your schemas there. Here’s an inline example
(m/validate ::order-id "123" {:registry (merge
                                         (m/default-schemas)
                                         {::order-id :int
                                          ::order-multiplier :int
                                          ::order-amount decimal?})})
#2022-08-1119:42valtteriMore info how to work with registries here https://github.com/metosin/malli#schema-registry#2022-08-1119:43dangercoderThanks a lot for the answer, cheers!#2022-08-1120:03dangercoderCool thing. I created a custom “register function”.
(s/register ::api-key [:string {:min 1}])
in IntelliJ/cursive i resolved my function as clojure.spec.alpha/def — this way I can jump to the schema-definition of a namespaced keyword if there is a register call. 😄
#2022-08-1120:03dangercoder🤯#2022-08-1120:27valtteriCool!#2022-08-1510:20pinkfrogHi, running this example I got sci not available
(m/decode
  [string? {:decode/string 'str/upper-case}]
  "kerran" mt/string-transformer)
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138). ; :malli.core/sci-not-available {:code str/upper-case}
#2022-08-1510:21borkdudeI think you should include SCI as an optional dependency yourself. If you don't use the quotes, you won't need SCI#2022-08-1510:39pinkfrogThanks. Dunno what’s the added benefits of sci.#2022-08-1510:40borkdudeThe benefit of SCI would be if you would like to serialize your schemas to the front-end#2022-08-1510:40borkdudeThe quote is only needed for serialization#2022-08-1510:42pinkfrogThen why not a custom export function to convert function to its symbol names. I am not quite getting the actual benefits compared of importing sci here.#2022-08-1510:43borkdudeBecause just a symbol won't give you evaluation in the client#2022-08-1510:46pinkfrogWhen client are you referring to the js environment? Also, pardon me, are we assuming sci could run on js (given the recent work of nbb and such)#2022-08-1510:46borkdudeyes, JavaScript#2022-08-1510:47borkdudeyou can include malli on the client side and then use SCI to evaluate your schemas and get exactly the same validation as on the server#2022-08-1510:49borkdudeSo you can send your schema's dynamically to the client#2022-08-1510:51pinkfrog> So you can send your schema’s dynamically to the client That is so futuristic, that I am not sure I am ready.#2022-08-1511:16valtteriIn the latest version of the README the examples have been changed to not require SCI unless mentioned.#2022-08-1511:17valtteriI think this particular one is now
(m/decode
  [string? {:decode/string clojure.string/upper-case}]
  "kerran" mt/string-transformer)
; => "KERRAN"
#2022-08-1510:40pinkfrogI wonder why https://cljdoc.org/d/metosin/malli/0.8.9/doc/readme#value-transformation:~:text=Adding%20optional%20keys%20too%20via%20%3A%3Amt/add%2Doptional%2Dkeys%20option%3A. here uses qualified keyword, ::mt/add-optional-keys, but others like :decode/encode does not use qualfied ones#2022-08-1512:09ikitommiMalli started with simple keywords everywhere, but as the options are passed everywhere, this doesn’t scale well -> moved to use qualified keywords. Could clean up all options for 1.0.0?#2022-08-1512:11ikitommiOne ideas was to have default global options map, could be read from EDN-file, set like the registry etc and which could be tune all different parts of the system. Not there yet.#2022-08-1513:37pinkfrogI’d like to learn more on the scale well part. When I am writing something only used internally, I will use qualified keywords, because this helps jump to usage, bulk rename, etc. But when I am providing some opts map for the user, I will use simple keyword. Mandating qualified keywords kinda complicates the opts map.#2022-08-1515:33ikitommiMalli options has :`registry` being the only reserved simple key. There are some qualified keys used in core like ::m/walk-entry-vals, which are relevant only in a special use case (here, using m/walk). If we had two option arguments ("how the system is configured" and "how this use case should work") we could use simple keys in both. Now, it's just one arg, and I think it's better to namespace the latter keys to avoid conflicts and to document which ns owns them. You can merge 'em all into one map and pass it everywhere.#2022-08-1515:35ikitommiI recall there are two open issues related to how to manage the global options better. Both would introduce a breaking change in the public APIs.#2022-08-1515:36ikitommiI would like to remove the "pass in optional options arg to every function", have ideas on that, but, need to think about it first.#2022-08-1515:36ikitommiHere is a good example on bad and good option naming: https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L332-L361#2022-08-1714:30afleckis there an “exclusive or” way of combining schemas? say I have
(def foo-schema [:map [:id string?] [:foo number?]])
(def bar-schema [:map [:id string?] [:bar number?]])

(def foobar-schema [:orn [:foo foo-schema] [:bar bar-schema]])
if I do
(me/humanize (m/explain foobar-schema {:id :k :foo 3}))
I get
{:id ["should be a string" "should be a string"],
 :bar ["missing required key"]}
but ideally I only want to see a single :id ["should be a string"] error. I guess put more abstractly I’m wondering if there’s a way to know “minimal number of problems that if fixed would make the value valid”
#2022-08-1714:43afleckhmm seeming like maybe a multi schema could potentially be helpful#2022-08-1816:47ikitommi:multi is good option here#2022-08-1719:52Colin P. HillIs this a bug, or am I misunderstanding something? :thinking_face: It looks like it's complaining about the :=> schema only having one child, when it clearly has two.
(defn test-fn
    {:malli/schema [:=>
                    [:cat :int]
                    [:int]]}
    [n]
    n)
=> #'user/test-fn
(require '[malli.dev :as dev])
=> nil
(dev/start!)
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:136).
:malli.core/child-error {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}
#2022-08-1720:08Noah Bogartlooks like it's discarding :int , and then discarding the empty vector too? what happens if you change it to int??#2022-08-1720:08Colin P. HillIdentical error#2022-08-1813:55Noah Bogartsorry to not be more help, that's super weird#2022-08-1813:57Colin P. HillVersion 0.8.4, in case anyone wants to see if they can reproduce it#2022-08-1813:59Noah Bogartcan you share the full stack trace? maybe that will help#2022-08-1814:04Colin P. Hill
#error {
 :cause ":malli.core/child-error {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}"
 :data {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}}
 :via
 [{:type clojure.lang.ExceptionInfo
   :message ":malli.core/child-error {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}"
   :data {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :=>, :properties nil, :children [[:cat]], :min 2, :max 2}}
   :at [malli.core$_fail_BANG_ invokeStatic "core.cljc" 136]}]
 :trace
 [[malli.core$_fail_BANG_ invokeStatic "core.cljc" 136]
  [malli.core$_fail_BANG_ invoke "core.cljc" 134]
  [malli.core$_check_children_BANG_ invokeStatic "core.cljc" 162]
  [malli.core$_check_children_BANG_ invoke "core.cljc" 154]
  [malli.core$__EQ__GT$reify__7246 _into_schema "core.cljc" 1694]
  [malli.core$into_schema invokeStatic "core.cljc" 1921]
  [malli.core$into_schema invoke "core.cljc" 1912]
  [malli.core$schema invokeStatic "core.cljc" 1982]
  [malli.core$schema invoke "core.cljc" 1963]
  [malli.core$function_schema invokeStatic "core.cljc" 2390]
  [malli.core$function_schema invoke "core.cljc" 2387]
  [malli.core$function_schema invokeStatic "core.cljc" 2388]
  [malli.core$function_schema invoke "core.cljc" 2387]
  [malli.core$_register_function_schema_BANG_ invokeStatic "core.cljc" 2398]
  [malli.core$_register_function_schema_BANG_ invoke "core.cljc" 2395]
  [malli.core$_register_function_schema_BANG_ invokeStatic "core.cljc" 2396]
  [malli.core$_register_function_schema_BANG_ invoke "core.cljc" 2395]
  [malli.instrument$_collect_BANG_ invokeStatic "instrument.clj" 41]
  [malli.instrument$_collect_BANG_ invoke "instrument.clj" 39]
  [malli.instrument$collect_BANG_$fn__15014 invoke "instrument.clj" 70]
  [clojure.core.protocols$iter_reduce invokeStatic "protocols.clj" 49]
  [clojure.core.protocols$fn__8238 invokeStatic "protocols.clj" 75]
  [clojure.core.protocols$fn__8238 invoke "protocols.clj" 75]
  [clojure.core.protocols$fn__8178$G__8173__8191 invoke "protocols.clj" 13]
  [clojure.core$reduce invokeStatic "core.clj" 6886]
  [clojure.core$reduce invoke "core.clj" 6868]
  [malli.instrument$collect_BANG_ invokeStatic "instrument.clj" 70]
  [malli.instrument$collect_BANG_ invoke "instrument.clj" 58]
  [malli.dev$start_BANG_ invokeStatic "dev.clj" 23]
  [malli.dev$start_BANG_ invoke "dev.clj" 15]
  [malli.dev$start_BANG_ invokeStatic "dev.clj" 20]
  [malli.dev$start_BANG_ invoke "dev.clj" 15]
  [user$eval15353 invokeStatic "scratch_16.clj" 2]
  [user$eval15353 invoke "scratch_16.clj" 10]
  [clojure.lang.Compiler eval "Compiler.java" 7194]
  [clojure.lang.Compiler eval "Compiler.java" 7149]
  [clojure.core$eval invokeStatic "core.clj" 3215]
  [clojure.core$eval invoke "core.clj" 3211]
  [clojure.main$repl$read_eval_print__9206$fn__9209 invoke "main.clj" 437]
  [clojure.main$repl$read_eval_print__9206 invoke "main.clj" 437]
  [clojure.main$repl$fn__9215 invoke "main.clj" 458]
  [clojure.main$repl invokeStatic "main.clj" 458]
  [clojure.main$repl doInvoke "main.clj" 368]
  [clojure.lang.RestFn invoke "RestFn.java" 1523]
  [nrepl.middleware.interruptible_eval$evaluate invokeStatic "interruptible_eval.clj" 79]
  [nrepl.middleware.interruptible_eval$evaluate invoke "interruptible_eval.clj" 55]
  [nrepl.middleware.interruptible_eval$interruptible_eval$fn__10823$fn__10827 invoke "interruptible_eval.clj" 142]
  [clojure.lang.AFn run "AFn.java" 22]
  [nrepl.middleware.session$session_exec$main_loop__10925$fn__10929 invoke "session.clj" 171]
  [nrepl.middleware.session$session_exec$main_loop__10925 invoke "session.clj" 170]
  [clojure.lang.AFn run "AFn.java" 22]
  [java.lang.Thread run "Thread.java" 831]]}
#2022-08-1814:16Colin P. HillIssue disappears when I bump to version 0.8.9, which I don't see any reason not to do, so I suppose it's immaterial#2022-08-1814:17Colin P. HillThough I don't see anything about it in the change notes so maybe it was accidentally fixed and could still use a regression test, idk#2022-08-1814:32Noah Bogarti bet this is it: https://github.com/metosin/malli/pull/690#2022-08-1814:32Colin P. Hillseems likely#2022-08-1814:34Noah Bogartglad upgrading is the solution, that's a very easy solution lol#2022-08-1914:26ikitommiHi, is there an example cljs-project with shadow-cljs + malli.dev instrumentation enabled? Starting to port a large app from spec to malli, interested in good practices on how to do dev/prod separation, pretty printing errors in html (for react error boundaries on dev errors + enabling sci-powered code editing on the browser).#2022-08-1914:30ikitommiGoals: • great DX • clear separation of dev & prod validation • tools for building dynamic schema systems • small bundle size#2022-08-1914:31borkdudefor SCI-powered code editing I would take a look at https://nextjournal.github.io/clojure-mode/#2022-08-2113:12ikitommithat's great. Not a "dependency + one-liner to get an editor", but awesome in features etc.#2022-08-2113:14ikitommiBut, is there a simple (read-only) clojure code formatter for html? Malli error pretty printing is basically a sequence of text blocks and code blocks.#2022-08-2215:42dvingoI started playing around with a setup but it's closed source at the moment. I'd like some more details on what is meant by bullet 2 - separation of dev and prod. and for bullet 3 what is dynamic schema? constructing schema at runtime? based on data? for point 4, the entry namespace in my project includes https://malli.dev which will increase bundle size. With shadow-cljs I think one way to solve this is with ns-aliases but I haven't confirmed this works. I fear you may have to use 2 different builds one for dev one for prod. I'm not sure about great DX - I haven't used the cljs instrumentation on a large project, I think https://github.com/CrypticButter/snoop may offer better DX but has tradeoffs of its own - using new syntax for defn mainly. The reason is that the intsrumentation macro produces large amounts of code (a set! call for every instrumented fn plus the code to filter every ns + fn in them) so on large codebases I can foresee this being prohibitive perf-wise. I've thought a little bit about a shadow-cljs server plugin/extension that would track which functions are instrumented and which are stale, which would have to happen per client (open browser window running the app). Then the macro would have access to this state to only output instrumentation code for stale functions. I'm not sure how feasible that is though, or how much time it would take, or how maintainable it would be.#2022-08-2215:43dvingo@U21QNFC5C I think has been using the cljs instrumentation - any insights?#2022-08-2215:47ingesolThis is funny, was about to post a couple of questions about friction with tooling 🙂#2022-08-2215:51ingesolMaybe a bit into the weeds, but there are 2 issues holding us back from a full rollout right now: • If you remove :malli/schema metadata from a function, it remains instrumented • We are relying on function identity, in a way that is not easily changed, and that is broken by the wrapping function added in -instrument. I think we have added the setup required, specifically: • In shadow init, call md/start! before mounting our reagent comp • In after-load, call md/start! This is supposed to be sufficient according to the docs, but nothing is unstrumented until we call md/stop!#2022-08-2216:22ingesolAlso, all those ..instrumented ns.here is not very nice when there are tens and hundreds of them. Should ideally be grouped or optional.#2022-08-2217:10ingesolhttps://github.com/metosin/malli/issues/744#2022-09-1516:57ingesolAnother issue posted on CLJS instrumentation https://github.com/metosin/malli/issues/752#2022-08-2000:25geraldodevI have a react form that has a input :type "date" and the stored value is a string "2022-08-01". How to check for valid date in malli with clojurescript ?#2022-08-2013:57pppaulanyone here had success using malli.swagger?#2022-08-2014:05pppaul
{:info {:title "Site Map"},
 :paths {"" {:get {:parameters {:query {:type "object",
                                        :properties {:email {:type "string"}}},
                                :path {:type "object",
                                       :properties {:uuid {:type "string",
                                                           :format "uuid"}},
                                       :required [:uuid]},
                                :formData {:type "object",
                                           :properties {}}},
                   :responses {403 {:description "User tried to access something their not allowed to",
                                    :custom/template "mvp/error_pages/403.html"},
                               401 {:description "Bad Auth/Cookie gets a redirect to login"}},
                   :produces ("text/html")},
             :post {:parameters {:query {:type "object",
                                         :properties {:email {:type "string"}}},
                                 :path {:type "object",
                                        :properties {:uuid {:type "string",
                                                            :format "uuid"}},
                                        :required [:uuid]},
                                 :formData {:type "object",
                                            :properties {:email {:type "string"},
                                                         :otp {:type "array",
                                                               :items {:type "integer",
                                                                       :format "int64"}}},
                                            :required [:email :otp]}},
                    :responses {403 {:description "User tried to access something their not allowed to",
                                     :custom/template "mvp/error_pages/403.html"},
                                401 {:description "Bad Auth/Cookie gets a redirect to login"}},
                    :produces ("text/html"),
                    :consumes ("multipart/form-data")}},
         "/login3{uuid}" {}},
 :basePath "/swagger"}
this is what i'm giving it
#2022-08-2014:06Noah BogartMaybe thread this discussion?#2022-08-2014:10pppauli'm using https://github.com/metosin/ring-swagger maybe this doesn't work anymore? (it's pretty old). i looked at some of the tests and my swagger objects look pretty similar to the ones in the tests. i get complaints from swagger-jsonfn saying it doesn't know how to convert my object into a swagger schema#2022-08-2014:11pppaullooking at swagger's docs https://github.com/OAI/OpenAPI-Specification/blob/main/versions/3.0.3.md#parameterObject i think malli.swagger is generating wrong data#2022-08-2014:13valtteriCurrently it produces OpenAPI 2.x https://github.com/metosin/malli#swagger2#2022-08-2014:19pppaulwell, malli.swagger output is throwing errors for the other library for swagger that metosin made. i don't know what to do#2022-08-2014:20pppaul
{
  "skipParam": {
    "name": "skip",
    "in": "query",
    "description": "number of items to skip",
    "required": true,
    "type": "integer",
    "format": "int32"
  },
  "limitParam": {
    "name": "limit",
    "in": "query",
    "description": "max records to return",
    "required": true,
    "type": "integer",
    "format": "int32"
  }
}
it's not outputting this style of object
#2022-08-2014:28pppauli guess i have to study the swagger docs and make my own transformer#2022-08-2014:34valtteriPlease file an issue and we will look into it. 🙂#2022-08-2014:41valtteriI'm not sure if Tommi has intended that malli.swagger should play along with ring-swagger#2022-08-2014:41pppaul😞#2022-08-2014:44valtteriI don't also understand why would both be needed. AFAIK they both generate swagger.json#2022-08-2014:44valtteriThat you can use to render swagger ui#2022-08-2014:47valtteriDifference being that ring-swagger can generate the json from plumatic schema / clojure.spec and malli.swagger does that for malli schemas#2022-08-2014:53valtteriUhhhmm.. Need to correct myself. Looks like malli.swagger produces "chunks" of swagger 2.0 compatible JSON, not the complete swagger doc.#2022-08-2014:53pppaulok, i did not know that#2022-08-2014:54pppaulit makes sense that malli swagger wouldn't produce the whole swagger doc, i just don't know what to do with it's output#2022-08-2014:54valtteriWhich kinda makes sense since Malli knows only about the shape of data. It doesn't know anything about route definitions and other stuff that goes into swagger spec#2022-08-2014:55valtteriAre you using reitit by any chance?#2022-08-2014:55pppaulyada#2022-08-2014:55valtteriBecause there's reitit.swagger#2022-08-2014:56pppauli was hacking on yada swagger to try to get it working with malli#2022-08-2014:56pppaulchanging over to reitit is a bigger challenge#2022-08-2014:57valtteriYeah, I see. I don't know how Yada swagger works but in reitit it works so that it constructs the full swagger.json from route data + schema definitions. For schemas it supports plumatic, clojure.spec and malli.. And when it's constructing the swagger, it checks which impl it is, generates the needed JSON chunks and places them into the swagger doc#2022-08-2014:58pppaulyeah, i used malli swagger to generate the params data, but it still wasn't enough#2022-08-2015:01valtteriYeah.. I'm trying to look if ring-swagger has extension points where you could jack in malli#2022-08-2015:10valtteriMhmhmh to my 👀 it looks like some serious hacking would be required#2022-08-2015:22pppauli think i just have to change the params code, i already have merged route + method params#2022-08-2015:22pppauli just don't know what to change the params to, seems like i need to build ref structures and flatten out nested maps or something#2022-08-2015:24pppauli poked at it for a few hours, and i think i rather just see about using reitit...#2022-08-2113:11ikitommihere too https://github.com/metosin/reitit/tree/master/examples/ring-malli-swagger#2022-08-2117:32pppaulthanks, I think that's exactly what I need to figure out how to fix yada#2022-08-2211:40eskosSwagger is specifically pre-OpenAPI, which does get confusing. The name change and a whole bunch of backwards breaking changes happened with the release of OpenAPI 3.0, so a good mental note is that you really shouldn’t expect one from another. This isn’t necessarily very helpful to the issue at hand, but still an unfortunate detail one should be aware of.#2022-08-2116:18Elirazhello! I'm trying to work with on my project#2022-08-2116:19Elirazand am getting the following issue:
The required namespace "malli.dev" is not available, it was required by "main/root.cljs".
"malli/dev.clj" was found on the classpath. Maybe this library only supports CLJ?
#2022-08-2215:43dvingoyea what ingesol said sounds right, there's some notes here too: https://github.com/metosin/malli/blob/master/docs/clojurescript-function-instrumentation.md#2022-08-2116:19Elirazanybody knows what the issue is?#2022-08-2116:46ingesol@eliraz.kedmi157 You need to use malli.dev.cljs for CLJS I think.#2022-08-2214:32plinslooks like malli does not accepts a set as a schema what would be the equivalent of (spec/valid? #{:a :b} :a) ?#2022-08-2214:33Noah Bogartis this just :enum?#2022-08-2214:34Noah Bogartone of n possibilities?#2022-08-2214:34plinslooks like so, thanks#2022-08-2312:52Chris O’DonnellDoes anyone know the status of https://github.com/metosin/malli/pull/545 ? Running into the limitations of inst? at work, and it would be great to have a standard solution.#2022-08-2814:30ikitommihighly needed, pushed up on the TODO to check it out.#2022-08-2814:54Chris O’DonnellThanks! Let me know if there's anything I can do to help.#2022-08-2613:49Setzer22Hi! We've just noticed that schemas using a registry don't work with methods from malli utils, is this intended? This works as expected:
(m/entries [:map [:hola :string]]) ; => ([:hola [:malli.core/val :string]])
But this fails (returns nil, but the schema is the same)
(m/entries [:schema 
            {:registry {"Foo" [:map [:hola :string]]}}
            "Foo"]) ; => nil
#2022-08-2614:19Noah BogartYou need m/deref-all :
(m/entries
  (m/deref-all
    [:schema {:registry {"Foo" [:map [:hola :string]]}}
     "Foo"]))
;=> ([:hola [:malli.core/val :string]])
#2022-08-2614:24Setzer22awesome, thanks! 👍#2022-09-0112:39souenzzoHello In the context of this issue: https://github.com/metosin/malli/issues/739#issuecomment-1232938081 Should :my/int represent something different from [:my/int] ? or they both different notations with the same meaning? This behavior can be considered a bug?
(m/ast :my/int opts) #_#_=> {:type :malli.core/schema, :value :my/int}
(m/ast [:my/int] opts) #_#_=> {:type :my/int}
#2022-09-0112:43ikitommioh, doesn’t look right. works the same, but extra wrapping :thinking_face:#2022-09-0112:44ikitommineed to think about the issue, thanks for a gentle reminder 🙂#2022-09-0112:45souenzzothe extra wrapping makes one turn into $ref and another turn into a simple type, in json-schema generation.#2022-09-0112:46souenzzoI'm unsure if I should consider it a json-schema bug, or a malli bug.#2022-09-0122:44Justin ReedHow do I modify this:
[:map
   [:key-0 uuid?]
   [:key-1 string?]
   [:key-2 [:string {:max 9 :min 9}]]
such that a map must have one of these three keys? For example,
{:key-0 (UUID/randomUUID)}
{:key-1 "HELLO WORLD"}
{:key-2 "123456789"}
{:key-1 "HELLO WORLD" :key-2 "123456789"} 
are all valid, but
{}
{:key-4 "SOME OTHER KEY"}
are not. I didn't find a direct way of doing this in the docs, so I thought I'd ask before I ventured down the path of cobbling it together with :and, :or, etc.
#2022-09-0210:04iarenazaWhile it's not explicitly described as such, there is a hint about that possibility in the example shown in the paragraph with the text "Finding all subschemas with paths, retaining order:" There, the :fn schema is used to check that either the :streeet or the :lonlat keys are present in the :map schema (using the :and schema to tie both conditions). So something like this should work:
(mapv (fn [m]
        (malli/validate [:and
                         [:map
                          [:key-0 {:optional true} uuid?]
                          [:key-1 {:optional true} string?]
                          [:key-2 {:optional true} [:string {:max 9 :min 9}]]]
                         [:fn (fn [{:keys [key-0 key-1 key-2]}]
                                (or key-0 key-1 key-2))]]
                        m))
      [{:key-0 (UUID/randomUUID)}
       {:key-1 "HELLO WORLD"}
       {:key-2 "123456789"}
       {:key-1 "HELLO WORLD" :key-2 "123456789"}
       {}
       {:key-4 "SOME OTHER KEY"}])
#2022-09-0215:35emccue
[:or 
  [:map {:closed true}
    [:key-0 uuid?]]
  [:map {:closed true}
    [:key-1 string?]]
  [:map {:closed true}
    [:key-2 [:string {:max 9 :min 9}]]
  [:map {:closed true}
    [:key-0 uuid?]
    [:key-1 string?]]
  [:map {:closed true}
    [:key-1 string?]
    [:key-2 [:string {:max 9 :min 9}]]]
  [:map {:closed true}
    [:key-0 uuid?]
    [:key-2 [:string {:max 9 :min 9}]]]
  [:map {:closed true}
    [:key-0 uuid?]
    [:key-1 string?]
    [:key-2 [:string {:max 9 :min 9}]]]
  
#2022-09-0215:38emccueis my first go at it - obviously you'd write something that would produce that#2022-09-0217:32Justin ReedThank you both for your help. I was about to go down the road @U3JH98J4R demonstrated, but was hoping for something less redundant. @U8T05KBEW’s nudge is exactly what I was looking for.#2022-09-0403:35Jungwoo KimHi, is there any way to check var?
(m/validate [:map
               [:my-fn symbol?]] {:my-fn 'clojure.core/identity}) ;; obviously true
  (m/validate [:map
               [:my-fn var?]] {:my-fn (requiring-resolve 'clojure.core/identity)}) ;; expected true but invalid schema
As docs, https://github.com/metosin/malli#mallicorepredicate-schemas doesn’t support var? . How do I validate var? ?
#2022-09-0410:41respatialized[:my-fn [:fn var?]] If nothing else you could use a predicate schema#2022-09-0423:33Jungwoo KimAhh that’s so useful ! thank you!#2022-09-0807:04Ferdinand BeyerIt seems that I can't use recursive schemas in regular expression schemas?
(def selector-schema
  (malli/schema
   [:schema {:registry {::selector [:and vector? [:+ [:cat keyword? [:? [:ref ::selector]]]]]}}
    [:ref ::selector]]))

(malli/validate selector-schema [:foo [:bar]])
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/potentially-recursive-seqex [:ref :user/selector]
Is there another way to specify this? I basically want to specify a vector of keywords, that are optionally followed by a vector that conforms to the same spec...
#2022-09-0807:09ikitommiyou can’t inline the recursive references (reasoning https://github.com/metosin/malli/blob/master/src/malli/impl/regex.cljc), but you can use them, with explicit :schema wrapping, e.g.
(def selector-schema
  (malli/schema
   [:schema {:registry {::selector [:and vector? [:+ [:cat keyword? [:? [:schema [:ref ::selector]]]]]]}}
    [:ref ::selector]]))
#2022-09-0807:10Ferdinand BeyerThanks a bunch!#2022-09-0807:10Ferdinand BeyerThis works!#2022-09-0807:10ikitommithere was a discussion of auto-wrapping :refs here, but it might be more confusing.#2022-09-0807:10ikitommiYour welcome 🙂#2022-09-0807:11Ferdinand BeyerI wonder if we should place a hint to that somewhere in the README. Happy to give it a try and PR#2022-09-0807:11Ferdinand BeyerI was about to give up already, would never have guessed to wrap it in [:schema]#2022-09-0807:11ikitommi> As all these examples show, the “seqex” operators take any non-seqex child schema to mean a sequence of one element that matches that schema. To force that behaviour for a seqex child :schema can be used: >
(m/validate
>   [:cat [:= :names] [:schema [:* string?]] [:= :nums] [:schema [:* number?]]]
>   [:names ["a" "b"] :nums [1 2 3]])
> ; => true
> 
> ;; whereas
> (m/validate
>   [:cat [:= :names] [:* string?] [:= :nums] [:* number?]]
>   [:names "a" "b" :nums 1 2 3])
> ; => true
>
#2022-09-0807:12ikitommithat’s on the README, which is bit overgrown…#2022-09-0807:12Ferdinand BeyerOh man.#2022-09-0807:12Ferdinand BeyerSorry 🙂#2022-09-0807:12ikitomminp#2022-09-0807:13Ferdinand BeyerNote to self: Always read the whole section#2022-09-1210:34CarloCan I generate a value from a registry? Something like:
(def reg
  {::id int?
   ::name string?
   ::user [:tuple ::id ::name]})

(mg/generate {:registry reg} ::user)
#2022-09-1213:38Ferdinand BeyerYes, by wrapping it in [:schema]:
(mg/generate [:schema {:registry reg} ::user])
#2022-09-1213:54CarloThank you, could you expand a bit more on why this works/where to find docs?#2022-09-1217:40Ferdinand BeyerIt works since generate requires a schema to generate examples for. A registry is not a schema, just a map. So you need to create a schema that uses your registry. As https://github.com/metosin/malli#local-registry, any schema can have a local registry property, e.g. [:map {:registry ,,,}]. Since you don't have such a schema in your case, you can use the generic [:schema] one, and just refer to a named one from the local registry. An alternative would be the m/schema function to instantiate the schema explicitly. Remember that forms such as [:map] are not schemas yet, but will be instantiated on demand. You can explicitly turn them into schemas using m/schema. Hope that helps!#2022-09-1314:10CarloThat is much more clear, thanks for the thoughtful answer @U031CHTGX1T#2022-09-1307:04Martynas MIs there a decoder for malli schema itself from JSON? For instance if I have this schema: [:map [:key {:optional true} :string]] Can I parse it from this JSON?
["map",
	["key", { "optional": true }, "string"]
]
#2022-09-1310:04ikitommi[:map ["key" {:optional true} :string]] is also a valid schema, so one can’t guess from the JSON example that the "key" should be handled as a keyword.#2022-09-1310:05ikitommibut, if someone would write schemas of all malli schema syntaxes, one could do everything else (but not guess the string->kw things)#2022-09-1310:13Martynas MOk. That means that a dumbed-down version would work :thinking_face:#2022-09-1312:19ikitommiyes. If you want all maps to have keyword keys, you can: 1. read in from json 2. clojure-walk all vectors to have first arg as keyword 3. m/schema 4. m/walk and convert all map keys into keyword 5. profit#2022-09-1313:19Martynas MThere is also a difference between keyword and symbol . And that's also significant because :inst? ,`inst?` and "inst?" are different :thinking_face:#2022-09-1315:12ikitommimaybe Tagged JSON? https://github.com/metosin/jsonista#tagged-json#2022-09-1315:15Martynas MI was thinking about dumbing down the schema and making everything a keyword or a number :thinking_face: I don't need all power.#2022-09-1410:04juhoteperiIf you know all your map keys are going to be keywords, you could use walker and transform to change map keys to keywords from strings, in the read schema.#2022-09-1410:04juhoteperiOr you if just write limited Malli schema for the Malli schemas you need to support, you can use :keyword there to read the JSON strings as keys for the map keys.#2022-09-1410:39Martynas MI was thinking about making a subset of malli by doing something similar to a custom parser. And then use :inst instead of inst? and :number instead of number? to make it all very uniform for the reader and parser. And after parsing I would do clojure.walk/postwalk or something similar to replace into the functions that malli understands. So I should probably implement parsing for all basic types, :map , :set and some basic options like :optional. I think I don't need seq regex matching in this specific use as this will only be used to define a data structure and sequences, maybe lists with order too but there shouldn't be tuples.#2022-09-1314:16Anders EknertHey! Just discovered malli, and it looks like it does a lot of the things I need it for 😃 Could be I’ve missed something in the README, but one thing that isn’t immediately apparent to me, is if/how I can do validation of values that reference other attributes? So say that I have a map like:
{
  "min": 10,
  "max": 20
}
and I want to ensure that the min value is never greater than the value for max , etc
#2022-09-1314:58lreadThe malli playground is a great place to discover malli features. Maybe https://malli.io/?value=%7B%3Ax%201%2C%20%3Ay%202%7D&amp;schema=%5B%3Aand%0A%20%5B%3Amap%20%5B%3Ax%20int%3F%5D%20%5B%3Ay%20int%3F%5D%5D%0A%20%5B%3Afn%0A%20%20%7B%3Aerror%2Fmessage%20%22x%20should%20be%20greater%20than%20y%22%7D%0A%20%20(fn%20%5B%7B%3Akeys%20%5Bx%20y%5D%7D%5D%20(%3E%20x%20y))%5D%5D will give you inspiration?#2022-09-1315:00Anders EknertOoohh, that's very nice! Thanks very much 😃#2022-09-1315:06lreadYou are most welcome! I find it a great place to explore.#2022-09-1315:10Anders EknertFor sure! For whatever reason, I hadn't registered that was a thing before. Much appreciated 👍#2022-09-1409:01Anders EknertBack again 😅 While the custom validator function works for validation, I’ve found it breaks the default value transformer. My code looks something like this:
(def Conf
  [:map
   [:polling
    [:and
     [:map {:default {}}
      [:min-delay-seconds [int? {:default 60}]]
      [:max-delay-seconds [int? {:default 120}]]]
     [:fn
      {:error/message "max polling delay must be >= min polling delay"}
      (fn [{:keys [min-delay-seconds max-delay-seconds]}]
        (> min-delay-seconds max-delay-seconds))]]]])
Without the :and + :fn :
(m/decode Conf {} mt/default-value-transformer)

=> {:polling {:min-delay-seconds 60, :max-delay-seconds 120}}
And once that’s been added, the map comes back empty:
(m/decode Conf {} mt/default-value-transformer)

=> {}
I assume I’m doing something wrong, but I struggle with understanding what 🙂
#2022-09-1414:12lreadNot at my dev box right now, but maybe due to {:default {}}?#2022-09-1414:17Anders EknertDoesn’t make a difference I’m afraid… I’ve tried a large number of combinations, but apparently not the right one 😅#2022-09-1414:47lreadI don’t want to to think I know malli well at all, so there might be other ways. Moving the :default {} up seems to work:
(require '[malli.core :as m]
         '[malli.transform :as mt])

(def Conf
  [:map
   [:polling {:default {}}
    [:and
     [:map 
      [:min-delay-seconds [int? {:default 60}]]
      [:max-delay-seconds [int? {:default 120}]]]
     [:fn
      {:error/message "max polling delay must be >= min polling delay"}
      (fn [{:keys [min-delay-seconds max-delay-seconds]}]
        (> min-delay-seconds max-delay-seconds))]]]])

(m/decode Conf {} mt/default-value-transformer)
;; => {:polling {:min-delay-seconds 60, :max-delay-seconds 120}}

(m/decode Conf {:polling {:min-delay-seconds 1231}} mt/default-value-transformer)
;; => {:polling {:min-delay-seconds 1231, :max-delay-seconds 120}}
#2022-09-1414:50Anders EknertWow, it does! I must have tried pretty much any combination but that one facepalm Thanks again, Lee! #2022-09-1414:51lreadMy pleasure, more experienced mallites might chime in, but the above looks ok to me.#2022-09-1317:37marciolHey, I have some code that have dependencies on spec, something that I cannot address in the near term, so I still need to duplicate malli and spec schemas everywhere. I'm just wondering if there is a lib that allows some sort of translation of malli schemas into spec schemas, in a way that we can make other spec dependent code thinks that a malli schema is a spec schema. Does anyone knows about some think like this?#2022-09-1318:31pithylessI'm aware of: https://github.com/dvingo/malli-code-gen/blob/main/thoughts.md#clojurespecalpha https://github.com/dvingo/malli-code-gen/blob/main/src/main/space/matterandvoid/malli_gen/clojure_alpha_specs.cljc I remember reading the docs, but have no idea how complete the actual implementation is.#2022-09-1318:32marciolI will track this, can be a point where I can start. Thanks#2022-09-1318:38pithylessI also remember this presentation showing how to generate domain models from malli (similar to the above repo thoughts). It's not directly related to clojure.spec, but perhaps you may gleam some insights if you go down the road of actually implementing this kind of generator. https://www.youtube.com/watch?v=ww9yR_rbgQs#2022-09-1318:42pithylessI think both of these things came about before malli introduced https://github.com/metosin/malli#qualified-keys-in-a-map - perhaps with some clever use of macros and custom malli registries, you can get 80% there without much effort?#2022-09-1318:43marciolLets see, there is a lot of duplication going on so I need to solve it through some clever strategy#2022-09-1319:31dvingoYea, I paused working on that, but the good news is that since then malli added malli.core/ast which makes it a lot easier to parse and walk schemas. I have a transform for taking malli schemas describing a hashmap and producing pathom output vectors that is working: https://gist.github.com/dvingo/213633acfdd520bddcdc91fc1c7b9e44 you should be able to do something similar to output clojure specs. The Kondo output has a more complete set of schema types: https://github.com/metosin/malli/blob/master/src/malli/clj_kondo.cljc#2022-09-1319:32marciol🙌#2022-09-1319:33dvingoI think @U0A5V8ZR6 continued working on something similar https://github.com/Blasterai/malli-datomic/blob/master/src/blasterai/malli_datomic/spec_utils.cljc#2022-09-1321:28escherizeI want to use malli for the clj-kondo type-mismatch stuff on a large clojure project. But I’d rather not add it as a dependency. Is there a way to make malli.core/=> annotate functions in other namespaces? Then I can put all my m/=>’s into an un-committed namespace and still get the benefits of (). I looked at the code, and adding the feature doesn’t seem too tough. But I may be missing another approach#2022-09-1322:30dvingoyea that should be possible with a little custom code. Underneath => just calls -register-function-schema https://github.com/metosin/malli/blob/bf92680ad76e57697261dd45c52dd31ffa9a8e1e/src/malli/instrument.clj#L41 which takes an ns symbol, a name symbol, a schema and a metadata map#2022-09-1406:12ikitommimaybe m/=> should allow qualified symbols too? e.g.
(m/=> my.domain/foo ...)
#2022-09-1406:12ikitommiwould not be a big change I guess: https://github.com/metosin/malli/blob/master/src/malli/core.cljc#L2440-L2442#2022-09-1321:52escherizeHow do I make a schema for a vector that has 3 items: int, nil, string? e.g. [1 nil "a"] I can make a vector that is homogenous, or I can make a list using sequence schema. But I can’t figure this one out#2022-09-1404:22ikitommitry [:tuple :int :nil :string]#2022-09-1417:04escherizeExactly right! In hindsight I could have also:
(mp/provide
 [[1 nil "x"]
  [2 nil "y"]
  [1 nil "z"]]
 {:malli.provider/tuple-threshold 3})
#2022-09-1409:01Anders EknertBack again 😅 While the custom validator function works for validation, I’ve found it breaks the default value transformer. My code looks something like this:
(def Conf
  [:map
   [:polling
    [:and
     [:map {:default {}}
      [:min-delay-seconds [int? {:default 60}]]
      [:max-delay-seconds [int? {:default 120}]]]
     [:fn
      {:error/message "max polling delay must be >= min polling delay"}
      (fn [{:keys [min-delay-seconds max-delay-seconds]}]
        (> min-delay-seconds max-delay-seconds))]]]])
Without the :and + :fn :
(m/decode Conf {} mt/default-value-transformer)

=> {:polling {:min-delay-seconds 60, :max-delay-seconds 120}}
And once that’s been added, the map comes back empty:
(m/decode Conf {} mt/default-value-transformer)

=> {}
I assume I’m doing something wrong, but I struggle with understanding what 🙂
#2022-09-1605:39robert-stuttafordjust sanity-checking that this is appropriate - when using a registry, and having registered a spec for :my/keyword e.g. :string, then using that keyword in a :map, i simply do [:map [:my/keyword :my/keyword]] - there isn't some simpler syntax to use similar to clojure.spec.alpha/keys ?#2022-09-1613:52kennyI believe you can replace the vector with just the qualified keyword. e.g., [:map :my/keyword]#2022-09-1614:03robert-stuttafordsweet i'll try that!#2022-09-1605:40robert-stuttafordalso, i can use recursive :ref in a mutable registry, right? update: yes you can#2022-09-1619:06John MaruskaIs there a way to merge two constrained maps (`[:and [:map ...] [:fn ...]]`) together that doesn't drop the :fns of the second map? I haven't been able to figure it out so far, so (gen/sample-seq (mg/generator my-schema)) is giving results that fail m/validate#2022-09-1812:38ikitommi:thinking_face: with the latest master:
(mu/merge
 [:and
  [:map [:x :int]]
  [:fn 'map?]]
 [:and
  [:map [:y :int]]
  [:fn 'coll?]])
;[:and 
; [:map [:x :int] [:y :int]] 
; [:fn map?] 
; [:fn coll?]]
#2022-09-1812:39ikitommilooks correct @U8VJYTQ76?#2022-09-1915:01John MaruskaGuess I just needed the weekend break -- found the issue this morning inside the function in my :fn#2022-09-1811:29Ory BandHi. I'm trying ot use a default fn value transformer from the tips section (https://github.com/metosin/malli/blob/master/docs/tips.md#default-value-from-a-function) and also discussed here https://clojurians.slack.com/archives/CLDK6MFMK/p1626171933375100, but sci fails with "could not resolve symbol" with the function i'm using. Any idea why?
; (err) Execution error (ExceptionInfo) at sci.impl.utils/throw-error-with-location (utils.cljc:39).
; (err) Could not resolve symbol: get-num-cpus
#2022-09-1811:40Ory Bandgot a response from borkdude here, unclear how to proceed though - how to pass a namespace? https://clojurians.slack.com/archives/C015LCR9MHD/p1663501053285409#2022-09-1812:40ikitommiyou can pass in the normal sci-options via :malli.core/sci-options. what ever sci accepts, is accepted here too:
(m/validate
 [:fn '(fn [arg] (malli.impl.util/-invalid? arg))]
 ::m/invalid
 {::m/sci-options {:namespaces {'malli.impl.util {'-invalid? malli.impl.util/-invalid?}}}})
; => true
#2022-09-1812:41ikitommisci readme seems to have tips and helpers on how to bind multiple things at once.#2022-09-1812:41Ory Bandspecifically in my case, how can i add my custom fn get-num-cpu?#2022-09-1812:44ikitommi
(defn get-num-cpus []
  (-> (Runtime/getRuntime) .availableProcessors))

(m/validate
 [:fn '(fn [arg] (= arg (get-num-cpus)))]
 10
 {::m/sci-options {:namespaces {'user {'get-num-cpus get-num-cpus}}}})
; => true
#2022-09-1812:47ikitommidoes that solve the issue?#2022-09-1812:47ikitommibut, do you need sci here?#2022-09-1812:47Ory Bandthanks tommi! testing, getting back to you soon#2022-09-1812:47Ory Bandhmm, i thought i did. don't i? i need to calculate the value based on amount of available cpus#2022-09-1812:47ikitommiyou could also just remove the ' and say #(hash-map :thread-count (* 5 (get-num-cpus)))#2022-09-1812:48Ory Bandwait. i could do that? i thought you couldn't pass functions#2022-09-1812:48ikitommim/eval takes any function or a source code. for the latter, it uses sci to interpret it.#2022-09-1812:48ikitommiyou can’t store normal functions as data (over wire / db etc), but if you don’t need that, you don’t have to use sci.#2022-09-1812:49Ory Bandand i can put a fn under the :default key and it would work?#2022-09-1812:50ikitommiyes#2022-09-1812:50ikitommi
(m/validate
 [:fn #(= % (get-num-cpus))]
 10)
; => true
#2022-09-1812:50Ory Bandamazing. i'm going to test that and get back to you. another question: what if i need to calculate the :default based on another key's input?#2022-09-1812:50Ory Bandnot storing this as data#2022-09-1812:51Ory Bandsee this example, how can i avoid sci?
[:parallel-pull-count
      {:title       "Pull messages thread count"
       :description ""
       :default 1}
      pos-int?]
     [:ack-and-lease-extension-thread-count
      {:title       "ACK processing and lease extension thread count"
       :description ""
       ; immitate java-pubsub's default value
       ; 
       :default-fn '#(max 6 (* 2 (:parallel-pull-count %)))}
      pos-int?]]]])
#2022-09-1812:53ikitommicurrently, the transformation interceptors don’t see their parents, so, attaching a transformer into a map key -> just sees that key & it’s children. This could be changed, but would need an issue and some thinking on how to do that.#2022-09-1812:54ikitommibut, you can attach the default values into the :map itself -> in that case the default-fn sees the whole map.#2022-09-1812:54Ory Bandyeah but in case the map was given by the user and one of the keys is missing, i want to support defaulting there as well#2022-09-1812:56Ory Bandbtw, just tried your suggestion and it doesn't work:
:default #(hash-map :thread-count (* 5 (get-num-cpus)))}
i'm getting
; (out) clojure.lang.ExceptionInfo: Invalid configuration {"parsing_errors" {:concurrency ["invalid type"]}}
#2022-09-1813:04Ory Bandwhat am i doing wrong?#2022-09-1813:08ikitommithe :default key is defined in the default-value-transformer and it should be a value, e.g. (hash-map :thread-count (* 5 (get-num-cpus))). the default-fn-value-transformer defines a key :default-fn which can be a function, e.g. #(hash-map :thread-count (* 5 (get-num-cpus))).#2022-09-1813:10ikitommi
(m/decode
 [:map {:default {}}
  [:x {:default (get-num-cpus)} string?]
  [:y {:default-fn #(* (get-num-cpus) (:x %))} string?]]
 nil
 (mt/transformer
  (mt/default-value-transformer)
  (default-fn-value-transformer)))
; => {:x 10, :y 100}
#2022-09-1813:10Ory Bandso i should use default-fn then#2022-09-1813:10ikitommiI think so#2022-09-1813:10Ory Bandtesting#2022-09-1813:40Ory Bandit works. thank you!#2022-09-1819:50Anders Eknert> currently, the transformation interceptors don’t see their parents, so, attaching a transformer into a map key -> just sees that key & it’s children. This could be changed, but would need an issue and some thinking on how to do that. Interesting. That’s just the issue I was struggling with 😅 Should I create an issue?#2022-09-1820:03Anders EknertTo be more specific, I’m trying to get the name in the key of a :map-of construct, and use it as a default for one of the values in a sub-map#2022-09-2009:02hmadelaineHi everyone ! I have a question regarding malli.error/humanizeand a sequential schema. I have modified the example given in the README. Instead of having a nil latitude, I put a string.
(-> Address
      (m/explain
        {:id      "ID676"
         :tags    #{:artesan :coffee :garden}
         :address {:street "Ahlmanintie 29"
                   :city    "paris"
                   :zip    33100
                   :lonlat [61.4858322, "23.89"]}})
      (me/humanize))
=> {:address {:lonlat [nil ["should be a double"]]}}
I was expecting in the humanized error, this output :
{:address {:lonlat ["23.89" ["should be a double"]]}}
The result of m/explain indicates clearly in the errors that the input “23.89" is incorrect.
{:schema [:map
          [:id string?]
          [:tags [:set keyword?]]
          [:address [:map [:street string?] [:city string?] [:zip int?] [:lonlat [:tuple double? double?]]]]],
 :value {:id "Hiram",
         :tags #{:coffee :artesan :garden},
         :address {:street "Ahlmanintie 29", :city "paris", :zip 33100, :lonlat [61.4858322 "23.89"]}},
 :errors ({:path [:address :lonlat 1], :in [:address :lonlat 1], :schema double?, :value "23.89"})}
Is it normal ? Thnak you very much
#2022-09-2014:00ikitommithis is normal, for sequential, nil are used to mask the valid values, e.g. “second and fourth elements in error” = [nil nil ["error"] nil ["error"]]#2022-09-2014:00ikitommiyou can configure how to show the valid values, e.g.
(-> Address
    (m/explain
     {:id "Lillan"
      :tags #{:artesan "coffee" :garden "ground"}
      :address {:street "Ahlmanintie 29"
                :zip 33100
                :lonlat [61.4858322, "23.7832851,17"]}})
    (me/error-value {::me/mask-valid-values '...}))
;{:id ...
; :tags #{"coffee" "ground" ...}
; :address {:street ...
;           :zip ...
;           :lonlat [... "23.7832851,17"]}}
#2022-09-2014:22hmadelaineAlright ! Thank you very much 👍#2022-09-2014:51hmadelaineIt seems that me/error-value is not available in 0.8.9 ?#2022-09-2015:07hmadelaineDo you plan to release malli this month ?#2022-09-2015:47ikitommiyrs#2022-09-2015:48hmadelaineThank you for the excellent work 🙏#2022-09-2108:53hmadelaineHi ! With metosin/spec-tools I use the function (select-spec spec value) I did not find an equivalent in malli Do you plan to add it later or should I add it to my own ns malli.utils? Thanks#2022-09-2109:15ikitommithere is no helper, but:
(m/decode schema value (mt/strip-extra-keys-transformer))
#2022-09-2109:16hmadelaineYes that is what I did 😉 Thank you#2022-09-2109:16ikitommifor perf:
(def select-address (m/decoder Address (mt/strip-extra-keys-transformer)))

(select-address value)
#2022-09-2109:16ikitommiif you have both spec-tools and malli, please share perf numbers on that. I think it’s… big#2022-09-2109:17hmadelaineGood idea#2022-09-2216:57Ben SlessFound some performance improvement opportunities around parsing, especially around regex parsers. Also, trying to fold and/or parsers, but it's confusing 🙃#2022-09-2217:21ikitommilooking forward!#2022-09-2217:21ikitommito#2022-09-2217:25Ben SlessWould appreciate a pointer regarding what I'm getting wrong with orn this is the or parser:
(defn- -or->parser
  [children]
  (fn [f]
    (let [parsers (-vmap f children)]
      #?(:clj
         (reduce
          (fn [acc parser]
            (fn [data]
              (let [parsed (acc data)]
                (if (miu/-invalid? parsed)
                  (parser data)
                  parsed))))
          (fn [_] ::invalid)
          (reverse parsers))
         :cljs
         #(reduce (fn [_ parser] (miu/-map-valid reduced (parser %))) ::invalid parsers)))))
And the orn parser doesn't work
(defn- -orn->parser
  [this]
  #?(:clj
     (reduce
      (fn [acc [k _ c]]
        (let [parser (-parser c)]
          (fn [data]
            (let [parsed (acc data)]
              (if (miu/-invalid? parsed)
                (parser data)
                (miu/-tagged k parsed))))))
      (fn [_] ::invalid)
      (reverse (-children this)))
     :cljs
     (let [parsers (-vmap (fn [[k _ c]]
                            (let [c (-parser c)]
                              (fn [x] (miu/-map-valid #(reduced (miu/-tagged k %)) (c x)))))
                          (-children this))]
       (fn [x] (reduce (fn [_ parser] (parser x)) x parsers)))))
#2022-09-2406:07Ben SlessAnyway, first MR which I'm sure of submitted. Would you like to see before/after comparisons?#2022-09-2217:45Ben SlessI decided to break things apart and start with the improvements I'm sure of https://github.com/metosin/malli/pull/756#2022-09-2217:47Ben SlessOne radical thing I've considered trying is pivoting the values array in the Cache to as many arrays as CacheEntry has fields, avoids allocation and makes the entire thing cache friendly#2022-09-2218:02dvingoafter a comment by Thomas Heller (https://github.com/metosin/malli/pull/754#issuecomment-1252581659) I refactored the cljs function instrumentation https://github.com/metosin/malli/pull/755 to happen all in cljs with only collecting schemas happening in a macro. The benefits include the namespaces matching with the clojure versions (aside from malli.dev.cljs), a simpler mental model as instrumentation doesn't occur via generated code, and a more straight forward implementation that matches clojure's pretty closely.#2022-09-2220:17kennyWhen working with Malli and generators, how do folks typically handle lazy loading of test.check?#2022-09-2221:51dvingohttps://github.com/borkdude/dynaload is great for this sort of thing#2022-09-2301:08kennyCool lib :thumbsup::skin-tone-2: How do you set the generator on a schema to be dynamically loaded? i.e., if I'm setting :gen/gen key, it'd need to be loaded when setting it.#2022-09-2312:43dvingoi haven't done it myself, but perhaps walking the schema dynamically to produce a new schema? https://github.com/metosin/malli#programming-with-schemas the "Adding generated example values to Schemas" section in particular#2022-09-2409:51Ben Sless#2022-10-0707:54ikitommithis is awesome!!#2022-10-0708:37Ben SlessThank you! It's also a really good benchmark case for parsing, which is what led to my other adventures 🙂#2022-09-2511:06delaguardoHi! Is there a way to make a schema to validate a map has a set of keys and validate the rest of the keys against :map-of schema? This is very close to additionalProperties directive in json-schema.#2022-09-2511:45delaguardojust found this - https://github.com/metosin/malli/issues/43 discussion#2022-09-2511:10delaguardo
[:map {:extra-keys [:map-of {:min 1 :max 2} :string :string]}
  [:x [:= "X"]]
  [:y {:optional true} :string]]
I was thinking to add an option into -map-schema but I bit lost in the source code. Would appreciate any help to better understand how IntoSchema work internally 🙂 can’t find any low-level article about that
#2022-10-0707:53ikitommithere is a old issue about this. is important.#2022-10-0708:30delaguardoSo far I made a custom -map-schema with :extra-entries option where it expects to have a schema to validate the “rest” of the map
(defn -map-schema
  ([]
   (-map-schema {:naked-keys true}))
  ([opts] ;; :naked-keys, :lazy
   ^{:type ::m/into-schema}
   (reify
     m/AST
     (-from-ast [parent ast options] (m/-from-entry-ast parent ast options))

     m/IntoSchema
     (-type [_] :map)
     (-type-properties [_])
     (-properties-schema [_ _])
     (-children-schema [_ _])
     (-into-schema [parent {:keys [closed extra-entries]
                            :or {extra-entries true}
                            :as properties} children options]
       (let [entry-parser (m/-create-entry-parser children opts options)
             form (delay (m/-create-entry-form parent properties entry-parser options))
             cache (m/-create-cache options)
             extra-entries (if closed false extra-entries)
             extra-entries (cond (boolean? extra-entries)
                                 (m/into-schema (if extra-entries :any 'empty?) nil nil options)
                                 (vector? extra-entries)
                                 (m/schema extra-entries options))
             ->parser (fn [this f]
                        (let [keyset (m/-entry-keyset (m/-entry-parser this))
                              parsers (cond->> (m/-vmap
                                                (fn [[key {:keys [optional]} schema]]
                                                  (let [parser (f schema)]
                                                    (fn [m]
                                                      (if-let [e (find m key)]
                                                        (let [v (val e)
                                                              v* (parser v)]
                                                          (cond (miu/-invalid? v*) (reduced v*)
                                                                (identical? v* v) m
                                                                :else (assoc m key v*)))
                                                        (if optional m (reduced ::m/invalid))))))
                                                (m/-children this))
                                        extra-entries (cons (let [parser (f extra-entries)]
                                                              (fn [m]
                                                                (let [ks (keys keyset)
                                                                      m' (select-keys m ks)
                                                                      m (apply dissoc m ks)
                                                                      m* (parser m)]
                                                                  (cond (miu/-invalid? m*) (reduced m*)
                                                                        :else (merge m' m*)))))))]
                          (fn [x] (if (map? x) (reduce (fn [m parser] (parser m)) x parsers) ::m/invalid))))]
         ^{:type ::m/schema}
         (reify
           m/AST
           (-to-ast [this _] (m/-entry-ast this (m/-entry-keyset entry-parser)))

           m/Schema
           (-validator [this]
             (let [keyset (m/-entry-keyset (m/-entry-parser this))
                   validators (cond-> (m/-vmap
                                       (fn [[key {:keys [optional]} value]]
                                         (let [valid? (m/-validator value)
                                               default (boolean optional)]
                                           (fn [^Associative m] (if-let [map-entry (.entryAt m key)] (valid? (.val map-entry)) default))))
                                       (m/-children this))
                                extra-entries (conj (let [valid? (m/-validator extra-entries)]
                                                      (fn [^Associative m] (valid? (apply dissoc m (keys keyset)))))))
                   validate (miu/-every-pred validators)]
               (fn [m] (and (map? m) (validate m)))))
           (-explainer [this path]
             (let [keyset (m/-entry-keyset (m/-entry-parser this))
                   explainers (cond-> (m/-vmap
                                       (fn [[key {:keys [optional]} schema]]
                                         (let [explainer (m/-explainer schema (conj path key))]
                                           (fn [x in acc]
                                             (if-let [e (find x key)]
                                               (explainer (val e) (conj in key) acc)
                                               (if-not optional
                                                 (conj acc (miu/-error (conj path key) (conj in key) this nil ::m/missing-key))
                                                 acc)))))
                                       (m/-children this))
                                extra-entries (conj (let [explainer (m/-explainer extra-entries path)]
                                                      (fn [x in acc]
                                                        (explainer (apply dissoc x (keys keyset)) in acc)))))]
               (fn [x in acc]
                 (if-not (map? x)
                   (conj acc (miu/-error path in this x ::m/invalid-type))
                   (reduce
                    (fn [acc explainer]
                      (explainer x in acc))
                    acc explainers)))))
           (-parser [this] (->parser this m/-parser))
           (-unparser [this] (->parser this m/-unparser))
           (-transformer [this transformer method options]
             (let [this-transformer (m/-value-transformer transformer this method options)
                   ->children (reduce (fn [acc [k s]]
                                        (let [t (m/-transformer s transformer method options)]
                                          (cond-> acc t (conj [k t])))) [] (m/-entries this))
                   apply->children (when (seq ->children) (m/-map-transformer ->children))
                   apply->children (m/-guard map? apply->children)]
               (m/-intercepting this-transformer apply->children)))
           (-walk [this walker path options] (m/-walk-entries this walker path options))
           (-properties [_] properties)
           (-options [_] options)
           (-children [_] (m/-entry-children entry-parser))
           (-parent [_] parent)
           (-form [_] @form)

           m/EntrySchema
           (-entries [_] (m/-entry-entries entry-parser))
           (-entry-parser [_] entry-parser)

           m/Cached
           (-cache [_] cache)

           m/LensSchema
           (-keep [_] true)
           (-get [this key default] (or (m/-get-entries this key default)
                                        (and extra-entries
                                             (m/-get extra-entries key default))))
           (-set [this key value] (m/-set-entries this key value))))))))
#2022-10-0708:31delaguardoHowever it is incomplete and needs adjustments#2022-09-2615:12erwinrooijakkersIs there a way to describe that a map should contain either one specific key or another key? e.g.
{:foo 1} ; correct
{:bar 1} ; correct
{} ; incorrect, should contain one of :foo or :bar
#2022-09-2615:31delaguardo
[:or [:map [:foo [:= 1]]] [:map [:bar [:= 1]]]]
#2022-09-2615:32erwinrooijakkersyep thanks, 🙂 but there are many more keys in the map so that would be loads of duplication, maybe there’s a way to do ir inside the map#2022-09-2615:38erwinrooijakkerse.g.,
user=> (m/validate [:map [:or [:foo :int]] [:bar :int]] {:quz 1})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
:malli.core/invalid-schema {:schema :x}
#2022-09-2616:01hmadelaineYou could use a :fn schema for that purpose : It is naive but it works
(def FooOrBar
  [:and
   [:map
    [:foo {:optional true} number?]
    [:bar {:optional true} number?]]
   [:fn
    {:error/message "Only one key must be present"}
    (fn [data]
      (= 1 (count data)))]])


(m/validate FooOrBar {:foo 1}) => true
(m/validate FooOrBar {:bar 1}) => true
(m/validate FooOrBar {:foo 1 :bar 1}) => false
(m/validate FooOrBar {}) => false
#2022-09-2619:59pithyless> but there are many more keys in the map so that would be loads of duplication The duplication can be resolved via composition, ala :merge , :union or even :select-keys. Remember also, that you may need to close the map schemas appropriately, if you do not want both :foo and :bar to be valid together. I would prefer :or over :and, since it is more likely to be generator friendly; but you may even consider a version using :multi with custom dispatcher that checks for the existence of certain keys.#2022-09-2914:26darnok@erwinrooijakkers Have you found a solution?#2022-09-2914:27darnokI have similar case, and I tried :and and :or and closing all maps. It doesn't work, because one of the specs won't pass the keys missing in another maps.#2022-09-2616:08robert-stuttafordhttps://twitter.com/RobStuttaford/status/1574430197079752706 (is it ok for me to paste a link to a tweet like this?)#2022-09-2620:16richiardiandreaThis is very nice, I wonder if we could do the same with cider...it would be very cool to be able to customize the jump to reference functionality#2022-09-2620:59DrLjótssonI am trying to get swagger-ui to work in reitit but am failing. I just want to double check if reitit + swagger-ui works when malli is used to spec the routes. The docs at https://github.com/metosin/reitit/blob/master/doc/ring/swagger.md say that "Reitit supports https://swagger.io/ documentation, thanks to https://github.com/metosin/schema-tools and https://github.com/metosin/spec-tools., i.e. malli is not mentioned.#2022-09-2707:02DrLjótssonI can confirm that swagger works with malli if reitit.coercion.malli/coercionis used.#2022-09-2809:21Martynas MWhat does this error mean? https://github.com/metosin/malli/blob/c0965e2b2b37ea1d81e6f4ece7c87f56fa90398a/test/malli/core_test.cljc#L1799 I get it when I try to have a malli schema to decode malli tuples. And well... it has to be recursive :thinking_face:#2022-09-2809:23Martynas MI fixed the error by not using :ref in the definition like this:
::malli-tuple-schema [:cat
                      [:= :tuple]
                      [:* ::malli-schema]
                      #_[:* [:ref ::malli-schema]]]
#2022-09-2813:38ambrosebsI think it means that any regex operation cannot contain a recursive schema. checking for :ref is a crude way of checking for recursion.#2022-09-2813:39ambrosebsI developed a more refined way to detect recursive specs in the generators namespace. perhaps using that here would be a nice enhancement.#2022-09-2813:41Martynas M> cannot contain a recursive schema Well if I remove the :ref it will still reference the ::malli-schema but it will copy it. So the schema works the same way as it would work with :ref#2022-09-2813:42ambrosebsAgreed. That's the part that I think could potentially be improved.#2022-09-2813:43ambrosebseg., the ;; A bit undesirable, but intentional: test a few lines down from the test you linked.#2022-09-2813:43ambrosebsis ::malli-schema recursive?#2022-09-2813:43Martynas MEverything there is recursive 😄#2022-09-2813:43Martynas M
::malli-schema [:or
                                [:ref ::malli-map-schema]
                                [:ref ::malli-set-schema]
                                [:ref ::malli-vector-schema]
                                [:ref ::malli-tuple-schema]
                                :keyword]
#2022-09-2813:44ambrosebsah. I'm confused, you said you "fixed the error", what does that look like now?#2022-09-2813:44Martynas MThis looks like [:* ::malli-schema] and it works without the error#2022-09-2813:45ambrosebsthanks. now I understand how crude this check is 🙂#2022-09-2813:46ambrosebsI thought it was an overapproximation on checking for recursive specs. but it only guards against very specific recursive ones.#2022-09-2813:46ambrosebsunless I'm missing something.#2022-09-2813:47Martynas MThis is a more basic example that passes the checker:
[:schema
 {:registry
  {::malli-field-options [:map
                          [:optional {:optional true} :boolean]
                          [:min {:optional true} :int]
                          [:registry {:optional true} ::malli-map-schema]],
   ::malli-map-schema [:cat
                       [:= :map]
                       [:* [:or
                            [:tuple :keyword [:ref ::malli-schema]]
                            [:tuple :keyword [:ref ::malli-field-options] [:ref ::malli-schema]]]]],
   ::malli-schema [:or
                   [:ref ::malli-map-schema]
                   :keyword]}}
 ::malli-schema]
#2022-09-2813:51ambrosebsok, and that one has no references/`:ref` directly on regex ops. can you post the full one that yields the error and requires a change?#2022-09-2813:55Martynas MI don't know. I can't find it#2022-09-2813:55Martynas MI'll probably be editing it more. Maybe I can find it later. But when I got that bug I was unsure what to do. So it's a tough one.#2022-09-2813:56ambrosebsnp, just sounds like a good test case. I get the gist of it.#2022-10-0707:52ikitommithe :ref can’t be used to expand/inline things into a sequential schema. you can use :refs, but need to wrap it into :schema - takes just one position in the sequence. This is an implementation decision, described in the ns. If there is a performant way not to do this, I’m all 👂s!#2022-09-2820:40DiegoGot a question about malli’s :re schemas. I’ve got this code for validating email addresses:
(def email-address-regex  #"(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])")
(def email-address? [:re {:error/message "Please provide a valid email address"} email-address-regex])

(me/humanize (m/explain email-address? "
It seems like :re is using re-find because re-find does return true, but what can I do to make :re behave more like re-match ?
#2022-09-2909:02iarenazaYeah, it seems it's using re-find for validation: https://github.com/metosin/malli/blob/0.8.9/src/malli/core.cljc#L1350-L1351 In that case, you need to use the "beginning of line/string" (`^`) and "end of line/string" (`$`) anchors in your regular expression, so that re-find must match your regular expression over the whole input value, not just a part of it:
user> (require '[malli.core :as m])
nil
user> (require '[malli.error :as me])
nil
user> (def email-address-regex  #"^(?:[a-z0-9!#$%&'*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&'*+/=?^_`{|}~-]+)*|\"(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*\")@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:\
(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9]))\.){3}(?:(2(5[0-5]|[0-4][0-9])|1[0-9][0-9]|[1-9]?[0-9])|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])$")
#'user/email-address-regex
user> (def email-address? [:re {:error/message "Please provide a valid email address"} email-address-regex])
#'user/email-address?
user> (me/humanize (m/explain email-address? "
#2022-09-2919:54DiegoThanks!#2022-09-2919:05ingesolI tried to instrument a CLJS multimethod (the dispatch function), but it does not seem to work. I can call it with wrong input type, and it still works:
(defn my-dispatch
  {:malli/schema [:=> [:cat :string] :keyword]}
  [s]
  (keyword s)

(defmulti a-multi-method
  my-dispatch)

(defmethod a-multi-method :default [_]
  :the-default)

;; The below does not crash like it should
(a-multi-method 4)
#2022-09-2920:28ambrosebsJust a hunch, but have you tried:
(defmulti a-multi-method
  #(my-dispatch %))
#2022-09-3006:33ingesolThanks, that’s an interesting idea. The assumption is that the original function will be cached inside the multimethod storage? Anyway, it did unfortunately not work.#2022-09-3009:17ingesolAlso, to clarify, calling the my-dispatch function normally from somewhere else with wrong args triggers validation#2022-09-3013:30dvingoIt looks like this is due to how multimethods are implemented. https://github.com/clojure/clojurescript/blob/961807166c8cf4d45a225d63416f06464fb27eaf/src/main/cljs/cljs/core.cljs#L11343 the dispatch-fn is passed into a JS object (deftype) and that captures the original function value. Just tried this in JS:
var afn = function afn(){ console.log('FIRST')}
y = {x: afn}
afn = function (){ console.log('SECOND')}
y.x()
// => FIRST
so I think this is what the problem is
#2022-09-3013:32dvingowe may be able to mutate the MultiFn's dispatch-fn property when we instrument, but then the defmulti would have to have the instrumentation applied, not the function#2022-09-3014:36ambrosebs> Thanks, that’s an interesting idea. The assumption is that the original function will be cached inside the multimethod storage? Anyway, it did unfortunately not work. Yes, I was also going to suggest #'my-dispatch but I forgot how vars work in CLJS.#2022-09-3016:40ingesolTaking one step back, the most valuable thing for our codebase would be to be able to declare the contract for the multimethod in general. Instrumenting the dispatch function is more like a hack to get at least some of the value. I know that instrumenting defmulti is probably a large and maybe not attractive subject, just mentioning it.#2022-09-3016:55ambrosebsAFAIK it is not possible in CLJ.#2022-09-3016:56ambrosebsprobably easier in cljs#2022-09-3017:05ingesolTo illustrate what I mean
(defmulti say-hi
  {:malli/schema [:=> [:cat [:map [:nationality :keyword]]] :string]}
  (fn [person]
    (:nationality person)))

(defmethod say-hi :german
  "Guten tag")
#2022-09-3017:09ingesolI think I remember some earlier discussion indicating that the above is simplistic and probably not a good idea, but did not find it. And @U055XFK8V, just to clarify, this is the thing you are saying is not possible in CLJ?#2022-09-3017:47ambrosebsYes, I've looked at clojure.lang.MultiFn and there's no obvious way to achieve this.#2022-09-3017:47ambrosebsI managed to do this for defprotocol https://blog.ambrosebs.com/2022/09/08/schema-defprotocol.html#2022-09-3017:49ambrosebsactually, MultiFn isn't a final class so perhaps a wrapper can be created. I might look into that.#2022-09-3017:51ingesolBut that would require an alternative defmulti macro, right?#2022-09-3017:53ambrosebsI'm thinking we leave everything as-is, but an instrument! function when called on a multimethod would set the multimethod var to a thin wrapper around MultiFn.#2022-09-3017:53ambrosebsThe tricky part is making sure that wrapper works with existing defmethod calls. And if we subclass MultiFn, it might work.#2022-09-3017:54ingesolThis is all about the mechanics of doing the instrumentation, I also seem to remember there are issues with the semantics. Will try to look it up#2022-09-3018:00ingesolMaybe forget that last thing. So at least the way we use multimethods, it makes sense to enforce a shared input/output contract for all defmethods. The only thing that is not clear is how/where on the defmulti to define the schema.#2022-09-3018:00ingesolPinging @U055NJ5CC in case he has opinions on this#2022-09-3018:04ambrosebsThinking something like
public MyMultiFn extends MultiFn {
  public MyMultiFn(IFn wrapper, MultiFn inner) { }
  public Object invoke(args ...) {
    wrapper.invoke(inner, args...);
  }
....
}
(defn instrument-multi! [var checking-wrapper]
  (alter-var-root var (fn [multi] (MyMultiFn. checking-wrapper multi)))
#2022-10-0200:23ambrosebsI managed to prototype defmulti instrumentation successfully for Prismatic Schema, so I take back my assertion that this was impossible.#2022-10-0707:44ikitommiWould be great if multimethods could be schematized. No opinions on the impl.#2022-10-0718:44ambrosebsOk I'll prep a proof of concept#2022-10-0113:09Chris O’DonnellPut up a small PR fixing a typo I found testing the json schema implementation: https://github.com/metosin/malli/pull/757. Haven't contributed to malli before; please let me know if there's anything else I should do.#2022-10-0312:25robert-stuttafordis there a Correct™ way to test if a value is a valid malli spec? i'd like to avoid try catch if that's possible#2022-10-0313:36Noah BogartYou mean “can the given form be used as a Malli schema”, not “does the given form conform to an existing malli schema”, right? #2022-10-0314:23respatializedAFAIK there isn't yet a self-describing meta-schema for malli schemas; the only way I've done this is indeed with try/catch on m/schema.#2022-10-0314:25robert-stuttafordthat's right @UEENNMX0T 🙂 thanks @UFTRLDZEW!#2022-10-0707:42ikitommiSchemas can define their schemas as schemas. The IntoSchema has the needed hooks, someone just needs to write the Schemas for properties & Children.#2022-10-0403:35dumratRan into a little perf trouble with malli sampling. Looks like performance is exponential wrt to sample size. Is there a reason for this? Sample code:
(def cashflow-schema
  (m/schema
   [:map
    [:type keyword?]
    [:direction keyword?]
    [:incurred-by {:optional true} [:map [:type keyword?] [:name string?]]]
    [:amount int?]
    [:categories [:set string?]]
    ;; [:date inst?]
    [:description {:optional true} string?]
    [:towards {:optional true} [:map [:type keyword?] [:name string?]]]]))

(defn get-sampling-data [sample-sizes]
  (map (fn [ss] [ss ((comp first :mean)
                           (quick-benchmark (doseq [x (mg/sample cashflow-schema {:size ss})]
                                              x) {}))])
             sample-sizes))

(defn get-vis-data [sample-sizes]
  (let [data (get-sampling-data sample-sizes)
        values (map (fn [[ss time]] {:sample-size ss :time time}) data)]
    {:data {:values values}
     :encoding {:x {:field "sample-size" :type "quantitative"}
                :y {:field "time" :type "quantitative"}}
     :mark "line"}))

(get-vis-data (range 5 110 5))
(oz/view! *1)
#2022-10-0403:50Ben SlessCan you def a generator from the schema and run again?#2022-10-0405:27dumratYes, that works! Looks like it's almost constant time now.#2022-10-0406:15Ben SlessAwesome! All malli operations can instantiate single closures which already do all the heavy lifting and only need to validate, parse, w/e You should also try a larger range, and add a doall around the generated result,it may be lazy#2022-10-0406:39dumratNow that you mention it, it might be lazy, I agree. Time doesn't look right. But I tried generating samples with this and it's much faster than with previous method so for my purposes this is fine. Let me check the perf anyway#2022-10-0406:44dumratOne question: I used generator like this:
(defn generate-samples [sample-size]
  (let [gen (mg/generator cashflow-schema)]
    (doall (gen/sample gen sample-size))))
This is fine right?
#2022-10-0407:12Ben SlessYes, but you can even def the generator right under the schema. As you can see it does not depend on the sample size at all#2022-10-0407:21dumratYeah, so doing this makes it significantly faster (For example, to generate 500 samples, my previous code takes 10s while generator takes 1.5s so there's a marked perf improvement. But the growth is still exponential. But I think I will leave it there. I originally wanted to generate around 1000 samples and it took too long that's why I delved into this.#2022-10-0410:00Ben SlessThe old nerd snipe 🙂#2022-10-0403:36dumrat#2022-10-0508:22DenisMcQuick question about malli with swagger. Right now, I define my reitit route definitions with malli, and malli-swagger generates the swagger definitions - all working well. However, recently I have added a couple of API responses where the malli definition is of the form [:or [<option1 payload>] [<option 2 payload>]] In this case, swagger seems to just report <option 1 payload> as the route payload. How would I go about updating swagger so that it displays the options to the API user? Thanks in advance.#2022-10-0512:32eskosMaybe https://github.com/metosin/malli#multi-schemas would do the trick?#2022-10-0722:38DenisMcI’ll give it a go 👍#2022-10-0520:06dangercoderDoes anyone know an idiomatic way of making malli always generate UTF8 strings? I currently use my own :utf8-string type instead of :string#2022-10-0520:14Noah BogartWhen is it not generating utf-8 strings?#2022-10-0520:23dangercoder
(malli.generator/generate [:string {:min 10}])
#2022-10-0520:25dangercoderadding :min makes the generator not generate utf-8 strings consistently#2022-10-0611:56dangercoderhttps://github.com/metosin/malli/issues/758 Would be great if someone else could try it out and see if you get the same results#2022-10-0615:08ambrosebsI was the last person to touch that code so I should try it out...do you know how long this has happened and if there is a better generator to use?#2022-10-0615:41ambrosebslooks like this case is built on gen/char .#2022-10-0615:46ambrosebsperhaps gen/char-ascii or gen/char-alphanumeric should be used.#2022-10-0616:12ambrosebsproposed a fix https://github.com/metosin/malli/pull/759#2022-10-0817:33dangercoderThanks for the fix :heart_hands:#2022-10-0707:59ikitommihad a busy month and ending that now with a 1 week family vacation under a 🌴. Just kooked at the PRs, merged most, big thanks to all contributors! Will cut a release after a week.#2022-10-0710:36Noah BogartCongrats on the vacation! #2022-10-0708:18dharrigan🥳#2022-10-0819:45skynethow do I use the function generated from :malli/gen true in function metadata <https://github.com/metosin/malli/blob/master/docs/function-schemas.md#function-schema-metadata>? the doc says "Setting :malli/gen to true while function body generation is enabled with mi/instrument! allows body to be generated, to return valid generated data." for example
(require '[malli.core :as m])
(defn pow 
  {:malli/schema [:=> [:cat :int] :int]
   :malli/gen true} ; how do I find/use this generated function body?
  [x] 
  (* x x))

(require 'malli.dev)
(malli.dev/start!)
(pow 1)
;1
#2022-10-0913:43dvingoYou have to pass in malli.generator/generate to the start! call. there is an example under this section: https://github.com/metosin/malli/blob/master/docs/function-schemas.md#instrumentation ("With :gen we can omit the function body"..)
(defn pow {:malli/schema [:=> [:cat :int] :int] :malli/gen true} [x] (* x x))
(defn pow2 {:malli/schema [:=> [:cat :int] :int] :malli/gen false} [x] (* x x))
(md/start! {:gen malli.generator/generate})
(comment
  (pow 45)
  ; => random int

  (pow2 45)
  ; => 2025
  )
#2022-10-0916:52skynet@U051V5LLP perfect! just what I needed. didn't put it together to use the same argument to start!, thanks#2022-10-0917:47skynetoh and one missing piece: I now need to add :malli/gen false to the defn metadata in order to disable gen for specific functions when doing this, but this might work out#2022-10-0920:30dvingooh yea, I didn't realize that you have to opt out - another idea is to use the :filters option to target only those that you want to gen#2022-10-1110:05robert-stuttafordhas anyone had a go at writing malli specs with https://github.com/lambdaisland/regal ? what i like about this is that it makes the regex a little more self-documenting edit: nevermind, there's a malli section right in the readme!#2022-10-1615:30ikitommiI haven’t updated the Regal-side in some time. please update if the integration is out-of-date#2022-10-1615:31robert-stuttafordwill do!#2022-10-1118:54Nikolas PafitisHi I have a schema that looks like this
[:map 
  ...
  [:timeline-type [:enum :reactive :proactive-actual :proactive-alternative]]]
And I'm trying to use this schema with malli coercion to coerce the query params, now from the frontend i send this timeline-type as a string (obviously as it's in the query string) but it's not coerced to a keyword. Is there some option for :enum (or any other type for that matter) that I can specify a specific coercer for reitit. For example something like:
[:enum {:malli.coerce/coerce-as :keyword} :reactive :proactive-actual :proactive-alternative]
or something along those lines? Or what else could I do here.
#2022-10-1615:29ikitommiHi. There is an issue for resolving the real type from schemas, see https://github.com/metosin/malli/issues/264. Before that, you can use :and -> [:and :keyword [:enum :reactive :proactive-actual :proactive-alternative]]#2022-10-1119:39HankstenbergHi guys, quick question: If I run m/walk in clojurescript like this: (m/walk [:map [:id string?]] (m/schema-walker identity) I'm getting back a "#object[malli.core.t_malli$core35871]" instead of the vector I put in. What can I do with this object?#2022-10-1120:39dvingotry passing it to (malli.core/form to get the vector out if that's what you want#2022-10-1204:42HankstenbergOh, okay now I get it thanks!#2022-10-1204:36timothypratleyHello 👋 To include a literal, is it best to use :fn ? In these examples I want to limit the matches to a specific value. This seems to work:
((ma/parser [:fn #{1}]) 1)
;=> 1
((ma/parser [:* [:cat [:fn #{1}] [:fn #{2}]]])
 [1 2 1 2 1 2])
;=> [[1 2] [1 2] [1 2]]
Does this seem like the correct approach?
#2022-10-1212:49Chris O’DonnellI would probably use [:= 1] personally, but your approach seems fine, too.#2022-10-1420:57timothypratleyOh great thank you, I didn't see := that's what I was hoping for#2022-10-1605:18Ben Slessfn spec is best avoided IMO, it makes it hard to reason about specs#2022-10-1214:31Stig BrautasetCan I instrument protocol methods with Malli? We use protocols (implemented by records) quite a lot. With Spec we add an indirection via a regular function to have something to hang Clojure Spec specs off of. Is that what I’d do with Malli too?#2022-10-1317:15dvingoAmbrose was looking into this https://blog.ambrosebs.com/2022/09/08/schema-defprotocol.html https://clojurians.slack.com/archives/C06MAR553/p1661105488092049#2022-10-1712:31Stig BrautasetThank you!#2022-10-1218:14PrashantHi, I was curious if anyone has any pointers/implementation on validating EntityMap using Malli . These are returned by (datomic.api/entity db entity-id) . These EntityMap objects don't implement IPersistentMap so schema like [:map ...] fails with type mismatch. I would greatly appreciate the inputs.#2022-10-1302:05dvingoYou can create your own schema type using -simple-schema (here are some examples: https://github.com/metosin/malli/blob/1a9b3767f1d64d504663ca151363244db2635708/src/malli/core.cljc#L660) I'm not sure on the details but I think you can make your predicate function convert the entity to a hashmap and then leverage the existing :map schema type by having your custom schema type pass its arguments to the :map type#2022-10-1313:21PrashantThanks @U051V5LLP I will give it a shot.#2022-10-1314:06PrashantI have reached till below and was wondering how to pass the arguments to the :map type.
(defn entity?
  [entity]
  (instance? datomic.query.EntityMap entity))


(defn -entitiy-map-schema
  []
  (-simple-schema
    {:type :entity-map
     :pred entity?     
     :type-properties {:decode/map #(if (entity? %)
                                      (into {} %)
                                      %)}}))
On a sidenote, since a instances of datomic.query.EntityMap may have nested datomic.query.EntityMap, I think using a ::ref would also be needed?
#2022-10-1314:56dvingohmm, I think it might not be as simple as using -simple-schema you may have to copy the -map-schema implementation and delegate to that, here is a heavily hacked one I got working but gives the general idea:#2022-10-1314:56dvingo
(defn -entity-map-schema
  ([]
   (-entity-map-schema {:naked-keys true}))
  ([opts] ;; :naked-keys, :lazy
   ^{:type ::into-schema}
   (reify
     m/IntoSchema
     (-type [_] :entity-map)
     (-type-properties [_])
     (-properties-schema [_ _])
     (-children-schema [_ _])
     (-into-schema [parent {:keys [closed] :as properties} children options]
       (let [map-schema   (m/schema (into [:map] children))
             entry-parser (m/-create-entry-parser children opts options)
             cache        (m/-create-cache options)]
         ^{:type ::schema}
         (reify
           m/AST
           (m/-to-ast [this _] (m/-to-ast map-schema _))
           m/Schema
           (-validator [this]
             (let [keyset     (m/-entry-keyset (m/-entry-parser this))
                   _          (println "2 " (m/children map-schema))
                   validators (map
                                (fn [[key {:keys [optional]} value]]
                                    (let [valid?  (m/-validator value)
                                          default (boolean optional)]
                                      (fn [m] (if-let [map-entry (find m key)]
                                                (valid? (val map-entry))
                                                default))))
                                (m/children map-schema))

                   validate   (apply every-pred validators)]

               ;; THIS LINE IS THE SIGNIFICANT CHANGE:
               ;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;
               (fn [m] (and
                         (instance? datomic.query.EntityMap m)
                          (validate (into {} m))))))

           (-explainer [this path] (m/-explainer map-schema path))
           (-parser [this] (m/-parser map-schema))
           (-unparser [this] (m/-unparser map-schema))
           (-transformer [this transformer method options] (m/-transformer map-schema transformer method options))
           (-walk [this walker path options] (m/-walk-entries map-schema walker path options))
           (-properties [_] properties)
           (-options [_] options)
           (-children [_] (m/-entry-children entry-parser))
           (-parent [_] parent)
           (-form [_] (m/-form map-schema))
           m/EntrySchema
           (-entries [_] (m/-entry-entries entry-parser))
           (-entry-parser [_] entry-parser)
           m/Cached
           (-cache [_] cache)
           m/LensSchema
           (-keep [_] true)
           (-get [this key default] (m/-get-entries this key default))
           (-set [this key value] (m/-set-entries this key value))))))))
#2022-10-1314:57dvingothe main thing is when malli creates the schema (using -into-schema you take the provided children and create an underlying :map schema
map-schema   (m/schema (into [:map] children))
#2022-10-1314:57dvingothen fill in all the protocol methods for IntoSchema using that#2022-10-1314:58PrashantThanks a ton!!! this is super helpful.#2022-10-1314:58dvingothere may be a much simpler way to do this, I'm not an expert on this, just figuring things out by reading the source code of malli.core#2022-10-1315:00dvingoand you can try it with:
(m/validate
  (m/schema [:entity-map [:x :int]] 
    {:registry (assoc (m/default-schemas) :entity-map (-entity-map-schema))})
  (d/entity db [:some/id-prop 5]))
#2022-10-1315:00PrashantThis gives a very good head start. I was also thinking on the same lines of copying map schema and hacking through it.#2022-10-1315:01PrashantThanks again @U051V5LLP!!!#2022-10-1315:02dvingosure thing! I'm still learning myself, so would be curious what you come up with 🙂#2022-10-1615:23ikitommiWould it help if the malli.core/-map-schema took an extra option for the predicate? e.g. on could just do:
(def EntityMap (m/-map-schema {:pred entity?}))

(m/validate EntityMap ...)
#2022-10-1615:25ikitommiMalli is intended to be extendable, so all IntoSchemas are created using functions and it’s a easy & non-breaking change to add new options 🙂#2022-10-1615:26ikitommi(also, effects bundle-size on cljs, non-used schemas can be DCEd)#2022-10-1615:26ikitommianyway, PR welcome on adding the :pred for the m/-map-schema#2022-10-1714:10PrashantThanks for the suggestion Tommi. I will definitely look into implementing your suggestion and when satisfied, raise a PR 🙂#2022-10-1715:45Prashant@U055NJ5CC Raised this https://github.com/metosin/malli/pull/767 per your suggestion 🙂#2022-10-1406:33robert-stuttafordwhere can i learn about how to write generators for complex :multi specs? currently butting my head against this:
(mg/sample ;; OK
   [:map [:db/id {:optional true}]])

  (mg/sample ;; OK
   [:multi {:dispatch :type}
    [:some.type/value
     [:map
      [:some/key :string]]]])

  (mg/sample ;; Couldn't satisfy such-that predicate after 100 tries.
   [:and
    [:map [:db/id {:optional true}]]
    [:multi {:dispatch :type}
     [:some.type/value
      [:map
       [:some/key :string]]]]])
#2022-10-1410:28pithylessTwo things come to mind: 1. I think your :multi map needs to include the dispatch key (otherwise it won't be generated correctly) 2. Is it feasible for you to merge the multi map? This would work:
(mg/sample
    [:multi {:dispatch :type}
     [:some.type/value
      (mu/merge 
        [:map
         [:type [:= :some.type/value]]
         [:some/key :string]]
        [:map
         [:db/id :string]])]])
#2022-10-1410:29pithylessNevertheless, subscribing to this thread in hope that there is a better way to solve this problem (have encountered it as well).#2022-10-1410:32robert-stuttafordthanks that gives me some things to try 😅#2022-10-1415:39Stig BrautasetThis is a topic I’m interested in too. I tried to spec out the complete state of a board game (a toy project), and it was very easy to get into a situation where the generation failed.#2022-10-1507:21robert-stuttafordok so it seems i can't use mu/union because of cyclic dependencies between my specs; so it looks like generating data is off the table for me here. just in case there's something i'm missing, if you've any advice @U055NJ5CC, i'd greatly appreciate it 🙂#2022-10-1605:15Ben SlessYou can break cyclic dependencies with a ref spec#2022-10-1607:28robert-stuttafordi am using ref specs - the issue is that to use mu/union, the keyword specs you give it have to be registered already, which forces a specific code loading order, but i can't ensure that order because the spec is for an entity that links to other entities of the same spec, linked-list type 'next-item' / 'parent-item' stuff. the specs work as validation (with :ref specs); it just doesn't work to generate#2022-10-1607:32Ben SlessHuh, weird. Want to send the complete schema (or minimal repro)? I think I managed that once#2022-10-1607:49robert-stuttafordapex spec is :multi. all of the internal impls all a shared base :map spec, and some are that base plus some extra :map spec. the base :map spec has :refs to the apex :multi spec. then, after all that, i am also wrapping this in a generic Datomic id handling system, where whenever i specify a :db.type/ref, there's a wrapper spec that allows either a long, a tempid, a map with a long or tempid, or an :and of :map :db/id optional and whatever the actual spec was; in this case, the apex multi spec. all of this validates, but it doesn't generate 😬 i suspect what i want is conceptually not possible, but i don't know enough about malli or the domain it operates in to know for sure.#2022-10-1607:50robert-stuttafordi can provide pseudocode if that was too eye-watering to reason through 🙂#2022-10-1608:08Ben SlessHave you tried using a schema schema to wrap it all together?#2022-10-1608:08Ben Slesse.g. [:schema {:registry everything-you-just-said} ::apex]?#2022-10-1608:13robert-stuttafordah, i am incrementally building this up with calls to swap! on an atom that has been given to (mr/set-default-registry! (mr/mutable-registry *registry)) . i'll work up a minimal repro version of what i said without that and with [:schema {:registry ...} ::apex] and see how it goes!#2022-10-1615:17ikitommiHi, some comments on this: • :and generator generates on the first and just validates the generated result with the validators of the rest of the childs -> not likely to match • I was sure that there was an example on :multi + gen in README, but it seems there is not! but for now, there is no logic to merge the dispatch types into the schemas, so you have to do it yourself. Here’s an example from README:
[:multi {:dispatch :type
         :decode/string #(update % :type keyword)}
 [:sized [:map [:type [:= :sized]] [:size int?]]]
 [:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
• should find time to work on https://github.com/metosin/malli/issues/264, would help here and I think this is really important anyway • if all childs are maps and the :dispatch value is a Keyword or a String, there could be a helper to push the key + value into the maps. this helper could be used in gen, json-schema etc. namespaces to make it work correctly • I’m thinking :and is bad (like Plumatic Team found out) and there should be something like :constrained instead. • sorry @robert-stuttaford, I guess this didn’t help you, but that’s something for the background. Looking forward to your repro + Ben’s solution 🙂
#2022-10-1615:30robert-stuttafordthanks @U055NJ5CC, this is useful info!#2022-10-1616:53Ben SlessBased on some experience playing with kanren, should and implement a bind of one generator to the next?#2022-10-1617:12Ben SlessNot sure if it's viable, but a schema can describe how it can return a generator from a seed value to build up with bind. That way, for example, a map schema would use merge#2022-10-1615:17ikitommiHi, some comments on this: • :and generator generates on the first and just validates the generated result with the validators of the rest of the childs -> not likely to match • I was sure that there was an example on :multi + gen in README, but it seems there is not! but for now, there is no logic to merge the dispatch types into the schemas, so you have to do it yourself. Here’s an example from README:
[:multi {:dispatch :type
         :decode/string #(update % :type keyword)}
 [:sized [:map [:type [:= :sized]] [:size int?]]]
 [:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
• should find time to work on https://github.com/metosin/malli/issues/264, would help here and I think this is really important anyway • if all childs are maps and the :dispatch value is a Keyword or a String, there could be a helper to push the key + value into the maps. this helper could be used in gen, json-schema etc. namespaces to make it work correctly • I’m thinking :and is bad (like Plumatic Team found out) and there should be something like :constrained instead. • sorry @robert-stuttaford, I guess this didn’t help you, but that’s something for the background. Looking forward to your repro + Ben’s solution 🙂
#2022-10-1618:37ikitommi… and, it’s out 🥳. Thank’s everyone!!#2022-10-1721:38skynetI'm seeing a bug in malli 0.9.0 and 0.9.1, and I'm trying to get a minimal reproduction of this issue but I can't. the error I'm getting is:
Exception: java.lang.ClassCastExcetpion: class clojure.lang.Symbol cannot be cast to class java.lang.String (clojure.lang.Symbol is in unnamed module of loader 'app'; java.lang.String is in module java.base of loader 'bootstrap')
 at clojure.core$symbol.invokeStatic (core.clj:591)
    ...
    malli.instrument$_strument_BANG_$fn__24440$fn__24441.invoke (instrument.clj:23)
    malli.core$_instrument$fn__21304.doInvoke (core.cljc:2480)
    ...
    <my code>
this occurs in a test after doing !. there are some functions in the project with :malli/gen true in function metadata, and some with that set to false. malli is started with:
(malli.dev/start!
  {::m/function-checker mg/function-checker
   :report (pretty/reporter)
   :gen mg/generate})
this all works on 0.8.9. unfortunately I can't share the code, but I'm not sure this is enough for someone to fix it
#2022-10-1722:32skynetoh actually I figured it out, I was redefining a fn that was instrumented, and had it returning invalid data, like this https://github.com/skynet-gh/2022-10-17-malli-instrument-bug then malli calls (symbol n s) in that case https://github.com/metosin/malli/blob/master/src/malli/instrument.clj#L23 and n and s are symbols in that case, which doesn't work for multi-arity https://clojuredocs.org/clojure.core/symbol so changing it to (symbol (name n) (name s)) works#2022-10-1722:43skynetok I over-complicated it. I think it will break after ! with :report whenever you have any invalid input/output on an instrumented function#2022-10-1722:50dvingoOh shoot, I just touched that code recently, I can take a look#2022-10-1723:29dvingohttps://github.com/metosin/malli/pull/768 TIL - that code works in cljs !#2022-10-1813:24skynetthat's tricky, I didn't know that either#2022-10-1805:58ikitommi[metosin/malli "0.9.2"], thanks @danvingo for the quick fix!#2022-10-1805:59ikitommiwould be good to have tests for the dev-tooling too for the future.#2022-10-1812:51dvingoagreed! I will take a look into adding some for start!#2022-10-1822:15dvingo@ikitommi I'm writing some tests for ! and friends and noticed that the arguments are a little confusing. start! takes :ns which indicates the set of namespaces to be collected, but then instrument! doesn't use that set - instead you have to separately pass :filters with [(mi/-filter-ns...) do you think it makes sense to use the set of namespaces passed in to start! if they are present and pass those to filters so instrument! will use the same set? Otherwise you have to pass both :ns and :filters to only start collection and instrumentation on the desired set of namespaces#2022-10-1905:01ikitommi:thinking_face: just looked at the code, dev/start! doesn't seem to use any :ns option either. Where did you pick that up?#2022-10-1911:29dvingooh wow, that's my mistake! I see what happened - it's collect! that uses it, https://github.com/metosin/malli/blob/546eb663484e66ef1271b99189b17dbeca215ec8/src/malli/dev.clj#L23 which does support changing what is collected: https://github.com/metosin/malli/blob/546eb663484e66ef1271b99189b17dbeca215ec8/src/malli/instrument.clj#L46 I must have saw that and inferred that you wouldn't always want to collect everything when using dev/start! do you have any thoughts on if the :ns option should be supported for start!? I can change it back for the cljs version if you want#2022-10-2006:33ikitommiI think it should in start! too, but silly to be only on collect!#2022-10-1903:40JoelUsing the subschemas example, I wanted to pull out some custom data from the schema structure with the associated path. From something like this… [:street {:optional true :custom-field :value} string?] I want to make grab the path and that :value I put in that the properties. However, I see that using mu/subschemas does not return the properties. I guess I need to write my own walker? I find that puzzling since I think malli would need that :optional true info.#2022-10-1904:57ikitommicould you paste a minimal repro here?#2022-10-1913:44Joel
(def Schema
  (m/schema
   [:map
    [:purchases {:optional true } [:set string?]]]))
 (mu/subschemas Schema)
When :purchases is walked, it doesn’t list the {:optional…}
#2022-10-1914:37ikitommiIt’s because the properties are on the map-entry, not the child itself.#2022-10-1914:37ikitommithere is mu/find that works like clojure.core/find, e.g. returns the [key properties value] entry.#2022-10-1914:37ikitommiso, you can:
(def Schema
  (m/schema
   [:map
    [:purchases {:optional true} [:set string?]]]))

(defn get-in-props [schema path]
  (if-let [?entry (mu/find (mu/get-in schema (butlast path)) (last path))]
    (when (vector? ?entry) (second ?entry))))

(for [sub (mu/subschemas Schema)]
  (assoc sub :entry-props (get-in-props Schema (:path sub))))
;({:path [], :in [], :schema [:map [:purchases {:optional true} [:set string?]]], :entry-props nil}
; {:path [:purchases], :in [:purchases], :schema [:set string?], :entry-props {:optional true}}
; {:path [:purchases :malli.core/in], :in [:purchases :malli.core/in], :schema string?, :entry-props nil})
#2022-10-1914:37ikitommihope that helps.#2022-10-1914:38Joellooks like it will, ill give it a go.#2022-10-1914:44Joeli don’t understand what :malli.core/in is “about”, but this gets me what i need - thanks!#2022-10-1914:54ikitommiit's basically a pointer into to a homogeneous seq, "values in any position".#2022-10-1906:28timothypratleyHi 👋 I'm trying to understand 2 things about catn:
'[:catn
     [m [:schema [:ref "m"]]]
     [s [:schema [:ref "s"]]]
     [_ [:orn [v [:schema [:ref "v"]]] [k [:schema [:ref "k"]]]]]]
1. It seems to produce the same output whether I use :orn or :altn -- is there a reason to prefer one over the other? 2. The result it produces is {m {:a 1}, s #{:a}, _ [k :k]} is there any way I can produce {m {:a 1}, s #{:a}, k :k} instead? Where the catn takes the name from the choice operator (`:orn`)? Below is the full schema but you can ignore most of it, it's just the choice between vector and keyword at the end that I'm focusing on:
'[:schema
  {:registry {"start" [:and vector?
                       [:catn
                        [m [:schema [:ref "m"]]]
                        [s [:schema [:ref "s"]]]
                        [_ [:orn [v [:schema [:ref "v"]]] [k [:schema [:ref "k"]]]]]]],
              "m"     map?,
              "s"     set?,
              "v"     [:and vector?
                       [:catn [a [:schema any?]] [b [:schema any?]] [c [:schema any?]]]],
              "k"     keyword?}}
  "start"]
And an example input:
'{m {:a 1}
  s #{:a}
  k :k}
#2022-10-1920:08timothypratleyOne thing I discovered is that I can move the name binding "inside" cat using single argument orn instead of using catn:
[:cat
     [:orn [m [:schema [:ref "m"]]]]
     [:orn [s [:schema [:ref "s"]]]]
     [:altn [v [:schema [:ref "v"]]] [k [:schema [:ref "k"]]]]]
=> [[m {:a 1}] [s #{:a}] [k :k]] Which is interesting.
#2022-10-1923:18timothypratleyWhy does using a schema ref catn behave differently from an embedded catn:
(ma/parse
  '[:schema {:registry {"start" [:and vector? [:catn
                                               [a [:schema any?]]
                                               [b [:schema [:ref "b"]]]]],
                        "b" [:catn [c [:schema any?]] [d [:schema any?]]]}}
    "start"],
  '[1 (2 3)])

(ma/parse
  '[:schema {:registry {"start" [:and vector? [:catn
                                               [a [:schema any?]]
                                               [b [:catn [c [:schema any?]] [d [:schema any?]]]]]]}}
    "start"],
  '[1 2 3])
^^ The first (using a schema ref) matches a list inside a vector [1 (2 3)], whereas the second (embedded) matches a flat vector [1 2 3] -- I'm wondering if there is a way to use a ref to match a flat vector instead of a list.
#2022-10-2006:22ikitommitry to answer both here: • if you you a regex schema inside a regex schema, it get’s inlined, so you can create a “flat” list using those • non-regex schemas just take one slot in the regex schema, so: ◦ [:orn [:s :string] [:bs [:+ :boolean]] is “a string or 1+ booleans” taking 1 slot ◦ [:altn [:s :string] [:bs [:+ :boolean]] is the same but if used inside a regex schema, the :+ get’s inlined ◦ you can’t inline a :ref in the regex schema ◦ wrapping any schema into :schema makes it “normal schema” and thus, it takes just one slot in the regex (`:ref` and regex schemas)#2022-10-2020:30timothypratleyI see, thank you 🙂#2022-10-2020:05skynetwhen I'm using malli in a terminal like setting, I kind of want the pretty explain output reversed from how it is seen on a webpage: https://github.com/valyagolev/malli/blob/c7c22a8fefd3a8491a35e1e320ab620791c4500c/src/malli/dev/pretty.cljc#L37-L40 since the bottom is closest to what I'm looking at, and the top stuff I might have to scroll up to see. luckily I'm able to call this whole defmethod myself, but maybe it's common enough that other people would find it useful to have a function in that namespace to set explain to "console" mode?#2022-10-2106:55ikitommiconsole-mode sound good. Also, color themes (none/light/dark/custom) + html-emitting mode (for react error boundaries etc). How different would the :console mode be from the normal? All mm-methods should be different? just the one? some amount of config can be done via adding configuration options to the printer (it’s backed by a map), but if it’s totally different (the html-mode could be), then another mm & ns might be better.#2022-10-2113:12skynetI'm currently just rearranging the blocks (and removing the "more info" block):
(defmethod v/-format ::m/explain
  ...
    (-block "Schema:"
    (-block "Value:"
    (-block "Errors:"
only other thing I might change is the amount of whitespace. there's like 3 blank lines after the error, I might reduce to 1 line. mayyybe remove blank lines after the Schema:, Value:, Errors: section titles and content, but that could make it less readable
#2022-10-2106:55ikitommiconsole-mode sound good. Also, color themes (none/light/dark/custom) + html-emitting mode (for react error boundaries etc). How different would the :console mode be from the normal? All mm-methods should be different? just the one? some amount of config can be done via adding configuration options to the printer (it’s backed by a map), but if it’s totally different (the html-mode could be), then another mm & ns might be better.#2022-10-2112:16Thomas MoermanI'm currently implementing a custom schema (for reference types). My question: Is it idiomatic to call (m/validate ?schema ?subject {:registry ?my-local-reg}) within the body of a content-dependent schema like in:
(let [;; self-checking keyword spec is necessary to make `[:my/ref :asset/id] work
      local-registry
                 (merge
                   ;; self-validating schemas
                   {:asset/id   (m/schema [:fn #(= :asset/id %)])
                    :grommet/id (m/schema [:fn #(= :grommet/id %)])}
                   
                   ;; for ID values
                   {:or     (m/-or-schema)
                    :int    (m/-int-schema)
                    :string (m/-string-schema)
                    :uuid   (m/-uuid-schema)
                    :id     [:or :int :string :uuid]})
      
      ;; content-dependent schema
      Ref        (m/-simple-schema (fn [props children]
                                     (log/spy :info children)
                                     {:type :my/ref
                                      :pred (fn [x]
                                              (let [attr-schema (first children)
                                                    [id-attr id-val] (first x)]
                                                (and (map? x)
                                                     (= 1 (count x))
                                                     ;; ============== ;;
                                                     ;; Q: is this ok? ;;
                                                     ;; ============== ;;
                                                     (m/validate :id id-val {:registry local-registry})
                                                     (m/validate attr-schema id-attr {:registry local-registry}))))
                                      ;; Don't forget the children count constraints!
                                      :min  1
                                      :max  1}))

      custom-reg {:my/ref Ref}

      registry   (mr/composite-registry
                   m/default-registry
                   local-registry
                   custom-reg)]

  [(m/validate [:my/ref :asset/id] {:asset/id 123} {:registry registry})
   (m/validate [:my/ref :asset/id] {:grommet/id 123} {:registry registry})
   (m/validate [:my/ref [:or :asset/id :grommet/id]] {:asset/id 123} {:registry registry})
   (m/validate [:my/ref [:or :asset/id :grommet/id]] {:other/id 123} {:registry registry})
   ])
Is there perhaps another approach preferable? Thanks 🙏
#2022-10-2112:42Thomas MoermanI guess a better version would look somethat like -maybe-schema instead of using simple-schema#2022-10-2207:46ikitommiHave stumbled few times on writing custom reference schemas, not trivial. No best practices, but we have been adding new configuration options to schemas, (e.g. just added :pred to m/-map-schema), so if there is an existing schema type which is almost what you need and would be exactly what you need with adding an configuration option - I’m open to hearing & maybe adding that.#2022-10-2207:47ikitommibut, prefer using a cached m/validator instead of m/validate if possible, it’s much faster.#2022-10-2214:36Thomas MoermanYeah you're right, it's not trivial. I'm trying to get my head around the essence of what makes it so difficult, there is some kind of circularity in the problem, it seems.#2022-10-2207:42ikitommiShould m/-instument instrument also nested function definitions by default? https://github.com/metosin/malli/issues/770#2022-10-2400:53aaron51Does Malli have a way to wrap input in a vector, if it’s not already a vector? I know this is a one-liner in Clojure but it’s nice to keep all the coercion together. (if (vec? x) x [x])#2022-10-2404:50ikitommidon’t think so. what is your use case for this? input coercion?#2022-10-2404:51ikitommithere is mt/collection-transformer https://github.com/metosin/malli/blob/master/src/malli/transform.cljc#L416-L423, but doing just just any-seq-like->target-type, not wrapping non-seqs into collections.#2022-10-2410:40aaron51Use case is a reitit handler that accepts one or more than one multipart files. When the request has one file, the param is a map. When multiple files, the param is a vector. Would be a bit easier to handle if it was always a vector.#2022-10-2411:20ikitommiI see, you can create a custom transformer for this. like the mt/collection-transformer but with custom functions. You can map any transformer into reitit coercion. not fun, but doable.#2022-10-2614:35bortexzIs this a bug, or am I doing something wrong?
(defonce registry* (atom (m/default-schemas)))

(mr/set-default-registry! (mr/mutable-registry registry*))

(defn sdef
  "Defines a new schema in mutable [[registry*]]."
  ([type schema] (swap! registry* assoc type schema) schema)
  ([type props schema] (swap! registry* assoc type [:schema props schema]) schema))

(sdef ::thing-nested2 int?)
(sdef ::thing-nested (mu/optional-keys [:map ::thing-nested2]))
(sdef ::thing (mu/optional-keys [:map ::thing-nested]))

(mu/update ::thing ::thing-nested (fn [s] (mu/required-keys s [::thing-nested2])))
Blows up with:
; Evaluating file: malli.clj
; Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
; :malli.core/invalid-schema {:schema nil}
; Evaluation of file malli.clj failed: class clojure.lang.Compiler$CompilerException
#2022-10-2616:34bortexzIf reading the schema from the registry directly, it seems to work:
(mu/update (get @registry* ::thing) ::thing-nested (fn [s] (mu/required-keys s [::thing-nested2])))
#2022-10-2617:10ikitommihmm. that’s not good.#2022-10-2617:29ikitommiThe malli.util helpers do not deref the schemas => calling mu/get on a reference tries to get from the reference, not from the value behind it.#2022-10-2617:30ikitommicalling m/deref-all on the subject should work, but just for one level. This is unfortunate.#2022-10-2617:31ikitommichecked if that’s easy to change in malli, doesn’t seem to: https://github.com/metosin/malli/pull/772/files - all tests fail 😞#2022-10-2617:33ikitommihere’s the list of all paths that are part of the schema:
(mu/subschemas ::thing)
;[{:path [], 
;  :in [], 
;  :schema :malli.core-test/thing}
; {:path [0]
;  :in []
;  :schema [:map [:malli.core-test/thing-nested {:optional true} :malli.core-test/thing-nested]]}
; {:path [0 :malli.core-test/thing-nested]
;  :in [:malli.core-test/thing-nested]
;  :schema :malli.core-test/thing-nested}
; {:path [0 :malli.core-test/thing-nested 0],
;  :in [:malli.core-test/thing-nested],
;  :schema [:map [:malli.core-test/thing-nested2 {:optional true} :malli.core-test/thing-nested2]]}
; {:path [0 :malli.core-test/thing-nested 0 :malli.core-test/thing-nested2],
;  :in [:malli.core-test/thing-nested :malli.core-test/thing-nested2],
;  :schema :malli.core-test/thing-nested2}
; {:path [0 :malli.core-test/thing-nested 0 :malli.core-test/thing-nested2 0],
;  :in [:malli.core-test/thing-nested :malli.core-test/thing-nested2],
;  :schema int?}]
#2022-10-2617:34ikitommie.g. each reference schema contributes to the path with 0.#2022-10-2617:35ikitommiso, this would work too:
(mu/update-in ::thing [0 ::thing-nested] (fn [s] (mu/required-keys s [::thing-nested2])))
#2022-10-2617:36ikitommiplease write an issue, will think about this.#2022-10-2707:43bortexzDone (https://github.com/metosin/malli/issues/773) thanks for looking into this @U055NJ5CC 🙏#2022-10-2717:27souenzzodoes malli has something like spec's conform operation? my use case is: my schema has an or
:a string?
:b int?
:ab [:or :a :b]
And I a function that given the :ab schema + a value, it tells me if the value is :a, :b or :invalid
#2022-10-2717:35ikitommitry m/parse & m/unparse. If you want named nodes, you can: [:orn [:s :string] [:b :boolean]]#2022-10-2717:37ikitommisame with regexs, adding a n to the name, you can present the nodes in entry-style and get named results, e.g. :cat -> :catn, :alt -> :altn#2022-10-2717:47souenzzonice! TY#2022-10-2721:48mauricio.szaboFolks, can I ask for a huge improvement on Malli? Show the errors on instrument, on .pretty/explain, etc, at the end of everything#2022-10-2721:48mauricio.szaboSometimes when I get an error, I can't find it anywhere - either it is greater than my terminal buffer or it gets lost in the middle of a "your state is this" and "your schema is this"#2022-10-2721:57mauricio.szaboIt's not just inconvenient - on React-Native, for example, sometimes the console is not redirected to the REPL (because metro and other issues) so I keep having to disable and re-enable instrumentation 😢#2022-10-2814:26dvingothe reporter option can be passed in to instrument and to start, so pretty quick work to copy the pretty namespace to your project and mess around how you like#2022-10-2815:27mauricio.szaboI... actually have no idea what you mean 😄#2022-10-2815:27mauricio.szaboDo you have an example?#2022-10-2818:23dvingo
(malli.dev/start! {:report (fn [type data] (.error js/console "Instrument error: " (str type)))})
#2022-10-2818:25mauricio.szaboThanks, I'll try 🙂#2022-10-2818:27dvingosure thing 🙂 https://github.com/metosin/malli/blob/537c5fb8f96c35fc9e0d41629faa88a1aa8b56c3/src/malli/dev/pretty.cljc#L82 I meant you can copy this namespace and change it up as you like#2022-10-2723:02mauricio.szaboOk, for .pretty/explain there is a work-around - I can capture the :errors property and pretty-print them#2022-10-2809:57ikitommiAnyone interested in doing / sharing a Schema for def syntax? like the https://github.com/metosin/malli/blob/master/src/malli/destructure.cljc#L11-L55 here.#2022-10-2810:00ikitommiactually… this: https://github.com/metosin/malli/blob/master/src/malli/experimental.cljc#L8-L33#2022-10-2810:01ikitomminevermind, already working on it 🙂#2022-10-2814:24dvingoif you're touching that code, i think there is an edge case for cljs (maybe clj too) if you have a custom registry - the schemas used for parsing the experimental defn form may not be available - https://github.com/metosin/malli/pull/702/files#diff-8fa8701bc52f9f699e5512b8ea028a44c05033b134751af5f789cd57edcf05ad had to add that line to get it working.#2022-10-2815:15sashtonAre closed-schema and multi incompatible? I am calling closed-schema on a schema which includes a multi nested in it. It fails to validate correctly, but if I remove the closed-schema , it works as expected. Example:
(def operator-schema 
  (malli/schema
   [:multi {:dispatch :op}
    [:addition [:map
                [:summand-1 :int]
                [:summand-2 :int]]]
    [:multiplication [:map
                      [:factor-1 :int]
                      [:factor-2 :int]]]]))

(def equation-schema
  (mallu/closed-schema
   [:map
    [:left-hand-side operator-schema]
    [:right-hand-side :int]]))

(malle/humanize (malli/explain equation-schema {:left-hand-side {:op :addition
                                                                 :summand-1 1
                                                                 :summand-2 2}
                                                :right-hand-side 3}))

=> {:left-hand-side {:op ["disallowed key"]}}
#2022-10-2815:56sashtonNever mind, user error: I forgot to include the [:op :keyword] declaration in each of the submaps in operator-schema#2022-10-3109:49dumrat
(me/humanize
  (m/explain
    [:and
      [:fn (fn [{:keys [tag]}] (= :Portfolio tag))]
      ...]
    {:tag :some-other-tag ...}))
=>
["unknown error"]
I understand that the "unknown error" is the result because malli doesn't know what to say when my custom function returns false. Can I customize this error message here?
#2022-10-3109:50dumratI know I can use :enum here to do the same validation. Just want to know how to handle custom fns.#2022-10-3109:52ikitommi
(me/humanize
 (m/explain
  [:and
   [:fn {:error/message "horror"}
    (fn [{:keys [tag]}] (= :Portfolio tag))]]
  {:tag :some-other-tag}))
; => ["horror"]
#2022-10-3109:53dumratgreat thanks#2022-10-3109:53ikitommiyou’re welcome#2022-10-3115:12dumratData:
[{:key1 1 :val 4} {:key2 2 :val 5} {:key3 3 :val 6}]
I'd like to write a schema which requires all three keys key1, key2, key3 to be present in the sub maps of the input vector. How to do this?
#2022-11-0102:51dumratIs there a schema for malli schemas? 😄#2022-11-0105:29ikitommithere is support for building that 🙂 e.g. in IntoSchema protocol, there are:
(-properties-schema [this options] "maybe returns :map schema describing schema properties")
  (-children-schema [this options] "maybe returns sequence schema describing schema children")
… just have had no time to add the implementations to those. Validation of a form can be done with m/schema, throws if not a valid thing.
#2022-11-0116:11dumratAny reason why malli validation would not honor {:optional true} ? i.e.
[:map
 [:details
  [:map
   [:email {:optional true} :string]]]]
Proceeds to ignore the optional status when validating and fails validation.
#2022-11-0116:31timothypratley
(ma/validate [:map
              [:details
               [:map
                [:email {:optional true} :string]]]]
             {:details {}})
=> true
#2022-11-0116:32timothypratleyseems fine?#2022-11-0208:45dumratMy bad. I needed to use ‘maybe’ here.#2022-11-0208:47dumratIs there a way to use a schema which has few maybe schemas but consider maybe not be there for validation? Or do I have to strip the maybes from schema and do validation manually? On mobile, I can give code sample later if needed.#2022-11-0209:06ikitommiplease share an example#2022-11-0303:42dumratok case is like this. 1. I've got a few endpoints (around 20) returning large sets of xml. These are quite old services that we don't have control over. 2. I parse xml, then do some transformations on them 3. I have malli schemas for the transformed data. I coerce the data using schemas. Then I validate using same schemas. 4. Data from the services are often incomplete (nil values where they shouldn't be mostly). So I add [:maybe ...] for some fields to sort of make the validations work. 5. Now, I'd like to still know the cases where validations fail without the [:maybe ...] wrappings so I can have a report of missing stuff. Is it possible to do this or do I have to transform schema myself and remove the [:maybe ...] wrappings? sample:
(def counterparties-schema
  [:vector
   [:map
    [:address [:maybe :string]]
    [:baseNumber :int]
    [:contactName [:maybe :string]]
    [:contactPhone [:maybe :string]]
    [:cptyType [:maybe :string]]
    [:enabledForSettlement :boolean]
    [:id :int]
    [:legalVehicleId :int]
    [:mnemonic :string]
    [:name [:maybe :string]]]])

;; Validation with this succeeds
(m/validate counterparties-schema data)
=> true

;; But I'd still like to know fields where validation would have failed if [:maybe ...] wrappings weren't there.
;; i.e [:address :string] etc.

;; Manually something like this:
(defn strict [schema]
  (clojure.walk/postwalk 
   (fn [v] 
     (if (and (vector? v) (= (first v) :maybe))
       (second v) v)) 
   schema))

;; So I can still know the missing values
(m/validate (strict counterparties-schema) data)
What I was asking first is that is this kind of thing supported by malli itself? Perhaps that's a dumb question. Is this a common use case though? Or am I thinking about this wrong?
#2022-11-0209:07ingesolCLJS instrumentation reloading discussion in #clojurescript https://clojurians.slack.com/archives/C03S1L9DN/p1667377689417939?thread_ts=1667373586.775569&amp;cid=C03S1L9DN#2022-11-0217:39mauricio.szaboA question about decompleted map keys and values: if I register, for example, that :user/id is always an [:string {:min 1}], is there a way to query Malli for :user/id and know the result is [:string {:min 1}]? Or even if it's just a :string?#2022-11-0218:19ikitommi(m/schema :user/id) returns the ref, you can deref that with m/deref (or m/deref-all)#2022-11-0221:11mauricio.szaboThanks, I had to do one step further to be useful:
(m/ast (m/deref-all (m/schema :user/id)))
That helped 🙂
#2022-11-0308:17jeroenvandijkDid someone already use rewrite-cljc to point out malli validation errors inline? I believe it is possible and I’m thinking of doing this#2022-11-0309:14jeroenvandijkAll little bit how expound underlines invalid values with ^^^ (see https://github.com/bhb/expound#expound-1), so a little different than .pretty/explain i think. More like the output of babashka with for instance:
echo '(/ 100 0)' > foo.clj; bb foo.clj 

....
----- Context ------------------------------------------------------------------
1: (/ 100 0)
   ^--- Divide by zero
#2022-11-0309:40ikitommiI would also like to see those. Also, paiting the errors with pretty, like expound does it.#2022-11-0309:41ikitommidrafted mx/def on the way to ClojureDays, which could have a clj-kondo hook to highlight the errors in the editor: https://twitter.com/ikitommi/status/1585995979748409345. ping @borkdude#2022-11-0309:42borkduderewrite-cljc is now rewrite-clj maintained under org.clj-commons#2022-11-0309:42jeroenvandijkah yeah made a mistake rewrite-clj indeed#2022-11-0309:43jeroenvandijkHowever just found that I can probably do what I want with edamame as well#2022-11-0309:43jeroenvandijkNeeded this https://github.com/borkdude/edamame#postprocess for detailed location info#2022-11-0309:45ikitommidid a test with edamame a while back: https://gist.github.com/ikitommi/0e5c4e48d8aeb7dd176128856ecdacb5#2022-11-0309:48jeroenvandijkAh useful thank you!#2022-11-0309:49borkdude@ikitommi I think we could add "type checking" for def - this is what you need right?#2022-11-0309:54ikitommi@borkdude “type-checking” using clj-kondo type system, yes, that would be awesome!#2022-11-0309:55ikitommianother try would be use use malli for the validation and just emit errors. But that’s way more work I think (all the relevant schemas should be loaded, e.g. should need to run the whole program to figure our what is the schema)#2022-11-0309:55ikitommiso, should the mx/def emit clj-kondo types?#2022-11-0310:07borkdude@ikitommi when clj-kondo supports this yes, but currently I don't think it supports it yet#2022-11-0310:07borkdudelet's see if there is already an issue for it#2022-11-0310:10borkdudeah yes, a very old one :rolling_on_the_floor_laughing: https://github.com/clj-kondo/clj-kondo/issues/609#2022-11-0310:10borkdudeno upvotes ;)#2022-11-0310:15ikitomminow, one! 🙂#2022-11-0310:16ikitommibut, would that work here? the mx/def says “here’s a var which should have a value of type XYZ”#2022-11-0310:16borkdudethere is no support for this yet, I think#2022-11-0310:16borkdudeonly for functions#2022-11-0310:16borkdudeor perhaps there is#2022-11-0310:17ikitommicould hooks work here?#2022-11-0310:17borkdudemaybe try {:tag :int} or so#2022-11-0310:17ikitommi:thinking_face:#2022-11-0310:17borkdudeI have to go now, be back in an hour#2022-11-0310:17ikitommi:tag takes a clj-kondo type?#2022-11-0310:17ikitommisame here#2022-11-0310:17borkdudeyes: {:namespaces {foo {bar {:tag :int}}}} - if you are lucky this works, not sure#2022-11-0314:29jeroenvandijkFYI @ikitommi, I have something that I’m happy with (@borkdude also thanks to edamame), see https://gist.github.com/jeroenvandijk/080370966fadb7e65601931c3de47ed5#file-malli_inline_output-clj-L129. Here is some example output:
1: {:b :z
         ^------- should be an integer
  2: 
  3: :c {:c0 2}
              ^------- missing required key :x
  4: 
  5: }
     ^------- missing required key :a
#2022-11-0314:34lreadOh that's pretty neat @U0FT7SRLP. I might experiment with this strategy for reporting errors in cljdoc.edn files.#2022-11-0314:43borkdude@U0FT7SRLP Cool stuff, I think it's worth a blog post or maybe an article in the malli or edamame repo :)#2022-11-0314:48jeroenvandijkThanks @borkdude! Yeah I guess it can be useful for others, or in other contexts. Also it is combining some funny techniques together. Will think about an article. Do you have an example format you are thinking of?#2022-11-0314:49jeroenvandijkBtw, I have created a similar mechanism for giving feedback on invalid edn input, also using Edamame. Maybe can include this in the same article#2022-11-0314:50borkdudeI'm open to having an examples/malli/README.md thing in edamame for example, or similar, or doc/postprocess.md or so#2022-11-0314:50borkdudebut if you want to post it in your own blog, also fine, then we can link there#2022-11-0314:51borkdudeIf you don't have a blog, you can quickly create one with ... https://github.com/borkdude/quickblog :)#2022-11-0314:51jeroenvandijkhaha good one, let me think about it for a bit. No blog at the moment. I’ll come back to it#2022-11-0314:52borkdudeno pressure#2022-11-0315:11ikitommiLooks great! Also would like see both the code and the post 👍#2022-11-0315:13jeroenvandijkThanks @ikitommi a raw version is already here https://gist.github.com/jeroenvandijk/080370966fadb7e65601931c3de47ed5#file-malli_inline_output-clj-L129 (for the Malli feedback)#2022-11-0411:31borkdude@ikitommi https://twitter.com/borkdude/status/1588493830961393664 (WIP)#2022-11-0411:33borkdude(Need to fix a few test...
Ran 366 tests containing 3188 assertions.
80 failures, 8 errors.
)
#2022-11-0411:39ikitommioh, super! that was fast. I’ll emit the clj-kondo types when this ships. So, this would work with complex types like maps too and also for defining the values? e.g. “this var should hold a map of :age key with int value” and saying (def mything {:age "1"}) would be a failure?#2022-11-0411:39borkdudeI'll have to test it :)#2022-11-0411:54borkdudeGetting closer:
Ran 366 tests containing 3233 assertions.
1 failures, 0 errors.
#2022-11-0411:57borkdudeThis is obviously wrong facepalm#2022-11-0411:58borkdudeRepro:#2022-11-0412:02borkdude
Ran 366 tests containing 3233 assertions.
0 failures, 0 errors.
🎉
#2022-11-0412:04borkdudeStill getting false positives with test corpus... lunch time#2022-11-0412:06borkdudeIt's funny, I'm getting errors from dynamic vars which are initialized to nil and then later called with swap! - I guess I'll have to ignore dynamic vars#2022-11-0412:18borkdudehttps://twitter.com/borkdude/status/1588505774833426433#2022-11-0413:11borkdudeAnother nice edge case... https://github.com/clojure/clojurescript/blob/961807166c8cf4d45a225d63416f06464fb27eaf/src/main/cljs/cljs/core.cljs#L10803-L10816 It thinks gensym_counter is nil (since it was initialized to nil)#2022-11-0413:11borkdudeso swap! is giving a type error#2022-11-0413:51borkdude@ikitommi Now got this working:
$ clojure -M:clj-kondo/dev --lint - --config '{:linters {:type-mismatch {:namespaces {user {x {:tag {:op :keys :req {:x :any}} }}}}}}' <<< "(def x) (inc x)"
<stdin>:1:14: warning: Expected: number, received: map.
#2022-11-0413:51borkdudeSo here we declare that x is a map that has at least an :x key#2022-11-0413:54borkdudebut this is probably the reverse of what you were proposing. the config in clj-kondo is to override types that have been inferred, possibly incorrectly#2022-11-0414:00borkdudePerhaps clj-kondo should both check the type and take the config as the source of truth...?#2022-11-0414:00borkdudeI'm not sure how one would use that in practice#2022-11-0309:53JHi guys! Did you know if there a lib to convert a plumatic schema to a malli schema?#2022-11-0312:38jeroenvandijkNot sure if it exists, but maybe the lite syntax can help you convert your plumatic schema https://github.com/metosin/malli#lite#2022-11-0314:29jeroenvandijkFYI @ikitommi, I have something that I’m happy with (@borkdude also thanks to edamame), see https://gist.github.com/jeroenvandijk/080370966fadb7e65601931c3de47ed5#file-malli_inline_output-clj-L129. Here is some example output:
1: {:b :z
         ^------- should be an integer
  2: 
  3: :c {:c0 2}
              ^------- missing required key :x
  4: 
  5: }
     ^------- missing required key :a
#2022-11-0909:01JHi guys! I have a question about how to check “extra constraints”. Let’s say, we have this schema:
(def CreatCommentPayload
 [:map
  [:comment/owner uuid?]
  [:comment/text string?]
  [:comment/associated-account [:vector uuid?]]])
For example, this payload is valid for create a comment:
(validate CreateCommentPayload {:comment/owner #uuid "92688fe4-9050-42b0-af5d-6d74a3b3d65b"
                                :comment/text "Hello"
                                :comment/associated-account [#uuid "864fbd90-914d-4c6d-b1b9-415276f7dda8"]}
In fact, this is not enough validation. There is some other possible validations: • comment/owner should exist in the database. • Each account in comment/associated-account should exist in the database. There are several ways to check this two other facts: 1- Custom schema Like:
(def CreateCommentPayload
 [:map
  [:comment/owner [:and uuid? exists?]
  [:comment/text string?]
  [:comment/associated-account [:and [:vector uuid?] exists?]])
exists? is a custom function to check if one or several accounts is or not in the database. 2- Constraint schema Like:
(def CreateCommentConstraint
 [:fn (fn [comment-payload]
        (exists? (into (:comment/associted-account comment-payload []) (:comment/owner comment-payload))
(def CreateCommentSchema
 [:and CreateCommentPayload CreateCommentConstraint])
We reuse the exists?function but here we check all the accounts with one request. 3- Don’t check constraint in schema There will be a solution. The check of this constraints can be done in another layer of the app. What’s your position about this case?
#2022-11-0912:44Ben SlessSince the constraints you specify involve side effects, I'd say they shouldn't be in the schema#2022-11-0913:07eskosSchema is about structural validation, while you could with eg. fn schemas make them check from db all the things, it’s probably cleaner to implement that as separate validation layer. Or rely on db constraints themselves, and handle insert/update failures gracefully.#2022-11-1008:36JThanks @UK0810AQ2 and @U8SFC8HLP.#2022-11-0920:27dvingoAm I missing something obvious here?
(m/schema [:comment {:some :prop}] 
  {:registry (merge (m/default-schemas) {:comment :int})})
=> [:int {:some :prop}]
does not work:
(m/schema [:comment {:some :prop}] 
  {:registry (merge (m/default-schemas) {:comment [:map [:a :int]]})})
Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
:malli.core/invalid-schema {:schema [:map [:a :int]]}
#2022-11-0921:00aaron51Is a :fn schema of (.exists (io/file x)) appropriate in Malli? Given that “schema is about structural validation”? (https://clojurians.slack.com/archives/CLDK6MFMK/p1667999272184759?thread_ts=1667984513.081219&amp;cid=CLDK6MFMK)#2022-11-1017:50CaseyIs there a function in malli already that does something like (select-keys MySchema a-map) to make a map containing only the keys defined in the schema? I don't want to define a closed schema, but I also don't want to pass unknown keys further down#2022-11-1018:06ikitommithere is an example of this in Malli readme, see https://github.com/metosin/malli#value-transformation#2022-11-1018:06Casey(select-keys (mu/keys MySchema) value) does this#2022-11-1018:07ikitommie.g. me/strip-extra-keys-transformer, it's recursive#2022-11-1018:07CaseyAh! I see, for use with (m/decode)#2022-11-1018:31CaseyThan you @U055NJ5CC!#2022-11-1513:09agigaoIn the context of coercion - can we process data before we pass it to a schema nested one level deeper? For example:
(def file-metada
  [:map
   [:name string?]
   [:file file-content]])

(def file-content
  [:vector
   [:map
    [:id uuid?]
    [:name string?]]])
Actually read the file and transform data before passing to file-content for validation.
#2022-11-1513:25ikitommisomething I wrote.#2022-11-1603:24dumratI have been looking for something like this. I thought to myself "Surely malli must have some way to transform data - not just coerce". I will give meander a go.#2022-11-1606:25ikitommiplanning to add some helpers in Malli for this, e.g. a pre-build matcher, meander-transformer, visualization etc. Needs to play with this a bit to see what is the relevant reusable part first.#2022-11-1809:48agigaoThank you!#2022-11-1608:50Bart KleijngeldIs this expected behavior and if so, can someone point out what I'm misunderstanding? Explicitly stating to use the default schemas registry causes an error (and actually every merge I've done so far does):
(m/validate
  [:map
   [:x string?]]
  {:a 1})  ;; => false
(m/validate
  [:map {:registry (merge (m/default-schemas))}
   [:x string?]]
  {:a 1})  ;; => clojure.lang.ExceptionInfo: :malli.core/child-error {:type :enum, :properties nil, :children nil, :min 1, :max nil}
                 {:type :malli.core/child-error, :message :malli.core/child-error, :data {:type :enum, :properties nil, :children nil, :min 1, :max nil}}
Shouldn't this be equivalent?
#2022-11-1609:21ikitommicurrently, the property-registry doesn’t allow IntoSchema instances, just Schemas. Original idea was that you can only present serializable things. Not sure if this is a good constraint, as you can anyway have non-serializable things in schemas, e.g [:string {:gen/gen …generator-function-here…}]#2022-11-1609:22ikitommiPlease write an issue out of this.#2022-11-1609:22Bart KleijngeldI will. Thanks for the explanation#2022-11-1609:27Bart KleijngeldSee https://github.com/metosin/malli/issues/780 (If you have any title suggestion that will make this issue most findable/descriptive for you let me know)#2022-11-1612:57Bart KleijngeldI'm looking to express something like an :or schema, but instead of enumerating the arguments by hand I'd like to supply a collection of schemas. So something like:
[:any-of schemas-coll]
instead of
[:or schema-1 schema-2 ... schema-n]
Especially nice because registries tend to be regular maps, so you can just pass those in. Of course I can construct the :or schema above from a collection, but I would expect some nice idiom to be present. Is there something like this present? Or should I create a new schema type for this?
#2022-11-1615:01ikitommino idiom, children are currently varargs, not a single collection. But:
(into [:or] [:int :string])
; => [:or :int :string]
#2022-11-1615:03ikitommialso, there is the map ast:
(m/ast [:or :int :string])
; => {:type :or, :children [{:type :int} {:type :string}]}

(update *1 :children conj {:type :boolean})
; => {:type :or, :children [{:type :int} {:type :string} {:type :boolean}]}

(m/from-ast *1)
; => [:or :int :string :boolean]
#2022-11-1615:06Bart KleijngeldThanks!#2022-11-1713:16hanDerPederIs this a bug?
(m/validate :at.least/ten 8
            {:registry (merge
                        (m/default-schemas)
                        {:at.least/ten [int? {:min 10}]})}) ;; => true, but 8 < 10
#2022-11-1713:18hanDerPederI would expect the props from the vector to be passed to the schema.
(m/properties
 (m/schema :at.least/ten
           {:registry (merge
                       (m/default-schemas)
                       {:at.least/ten [int? {:min 10}]})})) ;; => nil
#2022-11-1713:28ikitommiint? is just a predicate, does not support properties, :int is a real schema. Try [:int {:min 10}] instead.#2022-11-1713:28ikitommiwould be nice to get a warning out of this, but not there yet.#2022-11-1713:30juhoteperi@U055NJ5CC predicate schemas probably support some properties, just not these#2022-11-1713:30hanDerPederaha, thanks 👍 this would imply that there is not a real schema equivalent of inst? , right? how come?#2022-11-1713:31ikitommithere is a separate ticket for making good time schemas, e.g. :time, :date , …#2022-11-1713:32ikitommibut, bumped into the missing :inst just yesterday, so, hear you.#2022-11-1713:32juhoteperi(defn -int-schema [] (-simple-schema {:type :int, :pred int?, :property-pred (-min-max-pred nil)})) and int? is (-simple-schema {:type 'int?, :pred int?})#2022-11-1713:33juhoteperiWe could register property-pred for predicates where it makes sense#2022-11-1713:33juhoteperi.... though I might prefer just removing property predicates completely#2022-11-1713:34hanDerPederI second that, speaking as a relatively new user coming from spec I expected int? and :int to be equivalent#2022-11-1713:35hanDerPederbut to end I would just like to say you guys but out some amazing libraries. i've learned a ton from reading your code. thanks!#2022-11-1713:38ikitommialso https://github.com/metosin/malli/issues/636#2022-11-1713:43hanDerPedershouldn't this also work?
(m/properties
 (m/schema :at.least/ten
           {:registry (merge
                       (m/default-schemas)
                       {:at.least/ten [:int {:doc "this number needs to be at least ten"
                                             :custom/attrib :tranmogrify
                                             :min 10}]})}))
ie, I expect schema properties to be returned, but is nil
#2022-11-1713:52hanDerPederaha, malli.core/deref#2022-11-1716:37ikitommim/coercer and m/coerce landed on master. Runs decode + validate, throws with explain result in case of error. Have copied those fns once too many times between projects.#2022-11-1811:15ikitommihttps://github.com/metosin/malli#coercion#2022-11-1809:49lepistaneHello I can't seem to figure out how to make humanize and explain give me reason fit for me. So i gotta a schema
[:map
 {:closed true}
 [:client [:fn #function[app.validation/fn--37864]]]
 [:sport [:fn #function[app.validation/fn--37866]]]
 [:stream [:fn #function[app.validation/fn--37868]]]
 [:timestamp [:fn #function[app.validation/fn--37870]]]]
All these are custom functions (more tight than regular predicate) And when i send gibberish to be validated i wanted to have human readable output (in logs)
(me/humanize (m/explain schema data))
But i get
{:sport ["unknown error"], :stream ["unknown error"]}
which i don't find very useful and i would like these to be more useful. is there anyway for these errors to be different? solved
[:fn {:error/message "AAA"}
             (fn [val] ...)))]
#2022-11-1812:09ikitommithought on https://github.com/metosin/malli/pull/782?#2022-11-1908:46ikitommiIf someone has mad skills or just passion for visualizing things with javascrip/plantuml/any-open-source, would appreciate tips or help: I would like to visualize both the malli transformer chains + the meander transformations. Did the https://www.metosin.fi/blog/transforming-data-with-malli-and-meander/#visualization in the blog post with Joint+, but it’s under commercial licence, so not valid for OS. Just did a http://www.plantuml.com/plantuml/uml/fPHHxz8m4C3V-oak-_i0wYH0JBGGn1XUUT74HrFM0wfRIxgEe8HFvdVmCzckb7MXQ8nls-_krj_jRblMXcdJL2L1QOJv3XzLemi4LaD6Y_euKXnr1cS2Bq1m2hGsMiYTkzqANPirgx17IvQ0mc0rlYEJ7h5NYAQ7VeX3TtyXGV4JqpiqNf6hYWbvPzaqWhSFTTt4BW7bKJFkIPv-yLKovLzRSiJ2AEq8urnZxLZ53f7RUyzZkMTAMxlHjIApagbk4-qAQL1lMO6mEXrShL1o3FCv37xzV0bFRrv_6A_U3c6n07XQioFxgZxlhUci51BffcJ5j_RorKQLV6RmP3m4Pm3lKkEpk6G9UgxGx1M_mC2og1GooBgIuIFfILBwNvpFFOlCBcRswB1fyh3_E6sWcRIJpDrEjItw_QFqEb78jTZjJPu_3zCdn7z5b0eUeWi8PTsgXQpkVORaUE9H_5PHiZyMX73duJx0UvLshtj9axyb-SMzPOMMG-2MPvSvaMGf8PrgGaawxNXrqkylpfacd3bPekJsl_OR with PlantUML, but the layout is kinda bad and doesn’t support nested fields properly. 🧵#2022-11-1908:48ikitommi#2022-11-1908:50ikitommi#2022-11-1908:53ikitommihere’s another try with PlantUML:#2022-11-1908:57ikitommilast http://www.plantuml.com/plantuml/uml/fPH1Rk8m68NtFiMlC0lGJ6O4QAGs88fKiML3fIwHYN_0RM8Znq6qb9Fr3ZrJRT8Qfp6jgkwIv_V7txCVZ0jDbIxpZ10Kcki7cCbIfGYqW2X9Y-q_oL15i2Vm6pXB82cquc9bNvTS5NfASwof6Gs8N6FURIYzccAU47LDUkIRq_l18TvGjK9T8JAPKouQb6N9sV6ZDet7ugJO2sE_LAdpQag4-0kZR37LKbc5CgQmi5fEDmG0dkRlsgz6ajhgnr1QcZ1hL8QU2urgILE4oMvZpLoABWo7q7vv_WEjY__NtSbb1qOZWDQKRkcTl5qRj4761IQBZARtnyU_2vcnWSQThaHxBW1NeVioSaO3wZutJ98Te6qqRnGYY4uZ7SCVXEw5eB_5lAzHHAQ9qRdD-i7DVh1RMr5XUXNR4jc8PjzPrQE4CuWLNwrrF7ghOPAuwZYNEovdzY_KMtSJ6Vrim2sVEEmFgebox6dDrFlAL9MrwQgNsTbDUyzxpI8X7dZJAMm8loVXq_Gz_K02HiWO1RF_Mwy0, code expression together:#2022-11-1908:59jeroenvandijkWhat about https://mermaid-js.github.io/mermaid/#/classDiagramUnfortunately, no experience yet and not sure if it is better than PlantUML.#2022-11-1910:42ikitommiThanks, but I don’t think Mermaid supports field-level mappings.#2022-11-1912:27dumrat@U055NJ5CC I tried graphviz dot (for the first time - so this could be improved upon I guess): https://bit.ly/3TVe9NU#2022-11-2016:56ikitommi@UC1DTFY1G that's actually quite good, bit wonky but best so far :man-bowing:#2022-11-1910:47ikitommimerged the :enum & := type-inferrer with transformers into master. So, this works now:
(m/decode
 [:map
  [:enum1 [:enum :kikka :kukka]]
  [:enum2 [:enum 'kikka 'kukka]]
  [:enum3 [:enum 1 2]]
  [:enum4 [:enum 1.1 2.2]]]
 {:enum1 "kikka"
  :enum2 "kikka"
  :enum3 "1"
  :enum4 "1.1"}
 (mt/string-transformer))
;{:enum1 :kikka
; :enum2 kikka
; :enum3 1
; :enum4 1.1}
#2022-11-1912:57rafalwHi, I've started using malli's function intrumentation for cljs projects, it works great but on error I'm getting "Warning" message in browser's console, is it posible to make it to be an "error" instead of "warning"?#2022-11-1917:22dvingocan you post a screenshot? and how are you starting the instrumentation? this is how they look with .cljs/start! https://github.com/metosin/malli/blob/master/docs/clojurescript-function-instrumentation.md#errors-in-the-browser-console#2022-11-2309:25rafalwwhen its reload (shadow-cljs detects some changes) it's an warning, when page is refreshed (for example F5) it's an error#2022-11-2316:33dvingoah, i think this is due to the way shadow-cljs loads cljs code - it uses eval and prevents exceptions from being thrown as they normally do I've found the following pattern works to get proper malli exceptions displaying: https://github.com/matterandvoid-space/todomvc-fulcro-subscriptions/blob/766d27be316c3f2ab6a23bd8db30932ec0601a4f/src/main/space/matterandvoid/todomvc/client/dev_entry.cljs#L20#2022-11-1915:42Dannyhttps://github.com/mfikes/cljs-bean/pull/94#2022-11-1923:01aaron51How can I humanize malli.instrument/instrument! failures? With large data and a large schema, it’s really hard to tell where the problem is from the invalid-input failure#2022-11-2000:15aaron51Maybe I need to pass my own “report” function (defaults to -fail!) that throws a humanized error?#2022-11-2000:29aaron51Seems like the report function doesn’t receive parameters that can be humanized…#2022-11-2009:06ikitommi
(mx/defn plus [x :- :int, y :- :int] (+ x y))

(mi/instrument! {:report (pretty/reporter)})

(plus "1" "2")
#2022-11-2009:06ikitommi#2022-11-2009:06ikitommireporter is a function to create it, takes a bunch of optional options.#2022-11-2009:13ikitommiif you want to throw the humanized (instead of just printing it):
(mi/instrument! {:report (pretty/thrower)})
#2022-11-2009:15ikitommi#2022-11-2016:01aaron51Awesome. Thank you!#2022-11-2009:10ikitommimalli.provider defaults now to creating real schemas (e.g. :int) instead of predicate ones (e.g. int?):
(mp/provide [{:a [1 2 3]
              :b "kikka"
              :c true}
             {:a nil
              :b "kikka"
              :c "true"}])
;[:map 
; [:a [:maybe [:vector :int]]] 
; [:b :string] 
; [:c :some]]
+ also the :some schema.
#2022-11-2107:57HankstenbergHi guys, how can I repeat an element of a tuple one or more times? E.g. I'd like to have a tuple: [:tuple :int :string] and only the string part should occur one or more times. When I try :repeat or anything, the strings get wrapped into a vector. Or is there a better way to get a heterogeneous list like [12 "a" "b" "c" ...]?#2022-11-2109:00ikitommitry: [:cat :int [:+ :string]]#2022-11-2109:01HankstenbergAh, thank you very much!#2022-11-2115:42jeroenvandijkIs there a way to extend a :multi like how clojure.spec allows you to extend a multispec?#2022-11-2115:43jeroenvandijkSo far I found that all the keys of the :multi need to be in the body at definition. I would like to add entries later.#2022-11-2115:44jeroenvandijkWith a mutable registry and some hackery I can add entries, but it is not very clean#2022-11-2115:56jeroenvandijkI’m looking for something like this where I can register the dispatch type outside of the multi
(let [schema [:schema
              {:registry {:x [:map [:hello :int]]}}
              [:multi {:dispatch (fn [x] [:ref (:type x)])}]]]
  (or (->
       (m/explain schema {:type :x
                          :hello 1
                          })
       (me/humanize))
      :ok)) ;=> ["invalid dispatch value"]
#2022-11-2116:03ikitommiYou can use references (strings or qualified keys) in multi body, like with map: [:multi {:dispatch :type} :domain/user :domain/order]#2022-11-2116:04ikitommie.g. just reg the impls into registry reference them with the key#2022-11-2116:04jeroenvandijkAh I see now. Didn’t try qualified keys#2022-11-2116:04ikitommithere is also an example of lazy registry, could be used too: https://github.com/metosin/malli#lazy-registries#2022-11-2116:06jeroenvandijkBut it’s not possible to leave the multi without the key entries, right?#2022-11-2116:07jeroenvandijkI want to extend my schema after the first definition#2022-11-2116:08jeroenvandijkSo something like
(def CloudFormation
  (m/schema
    [:multi {:dispatch :Type, :lazy-refs true}]
    {:registry registry}))
#2022-11-2116:08ikitommiI would be easy to ad a new key e.g. :`methods` into multi, that is used for the lookups. Would that work?#2022-11-2116:09jeroenvandijkYeah that would be perfect#2022-11-2116:10jeroenvandijksomething like
.. :methods (fn [] @*my-schema)
And then I would change *my-schema somewhere else, right?
#2022-11-2116:11jeroenvandijkI think this would more or less cover what multispec can do#2022-11-2116:12ikitommiPlease write a PR#2022-11-2116:12jeroenvandijkYep, thanks#2022-11-2214:50ikitommi@U0FT7SRLP I meant an Issue is enough 🙂#2022-11-2309:36jeroenvandijk@U055NJ5CC 😅 I actually tried to implement your suggestion by adding a :methods option. And it works, but after the schema is loaded it doesn't revaluate when the state/multimethod updates. Ill create an issue with an explanation and push what i have so far in a separate draft PR#2022-11-2311:06jeroenvandijk@U055NJ5CC I created an Issue https://github.com/metosin/malli/issues/786 and the :methods experiment is in https://github.com/metosin/malli/issues/787#2022-11-2209:00HankstenbergWhat is the best way to tell if a given string is a valid malli schema? As I understand it this is a two-step process... first I need to determine if it's valid edn data and in a second step I have to determine if it's a valid malli schema. There is a schema called "`:schema` " in malli.core/base-schemas , but how do I use it properly?#2022-11-2209:02ikitommivalid string like "[:enum :kikka :kukka]"?#2022-11-2209:04HankstenbergWell the idea is that the user provides a schema definition and I want to check if it's a valid malli schema, so something like "[:cat :int]", anything that can be turned into a schema with m/schema in the current context should be valid, everthing else I'd like to give proper error messages on.#2022-11-2209:05HankstenbergI'd like to humanize the output of "trying to use the input as a malli schema".#2022-11-2209:05ikitommiwithout humanizing:
(malli.edn/read-string "[:enum :kikka :kukka]")
; => [:enum :kikka :kukka]

(malli.edn/read-string "[:enumz :kikka :kukka]")
; =throws=> :malli.core/invalid-schema {:schema :enumz}
#2022-11-2209:06HankstenbergYea, but is there a meta-schema for a malli schema? I thought it was maybe malli.core/base-schema/:schema#2022-11-2209:09HankstenbergIt's got to be something that me/read-string does under the hood.#2022-11-2209:49ikitommithere is no Malli Schema for Malli Schemas. There is support for it (each schema can define it's supported properties & schema for children), but no-one has had time to actually describe those.#2022-11-2209:49HankstenbergOkay good to know, thanks! 🙂#2022-11-2408:52hanDerPederwhats the point of having -update and -comp in malli.core? -comp unrolls more arities than clojure.core/comp, so I'm guessing it's a perf thing. But -update is the same as 3-arity clojure.core/update.#2022-11-2409:06ikitommiyeah, -comp is a perf thing, -update… can’t recall. looks like it’s not that useful atm.#2022-11-2414:21vemvHave you seen this error message while running a Malli-based codebase through Cloverage?
No implementation of method: :-type of protocol: #'malli.core/IntoSchema found for class: clojure.lang.PersistentList
don't know how to fix that
#2022-11-2414:29vemv...I fixed it by commenting out these lines we had: ^{:type ::m/into-schema} that pattern was sort of mindlessly copied from the Malli impl. What does it intend to do? Is it safe for us to remove?#2022-11-2416:50ikitommiit's safe to remove. It's only used for pretty printing the the reified this.#2022-11-2417:15vemvthanks! out of curiosity, do you have a sense of why that particular error is triggered? Does overriding ^:type have some sort of code-evaluation semantic to the clj compiler? Otherwise :type ::m/into-schema seems fairly 'inoccuous' to me, I don't see how it can trigger this issue#2022-11-2521:13dharriganBetter here I think: Got a small PR for consideration #2022-11-3014:20dharriganThank you @U055NJ5CC for merging in this small PR. I hope others may find it benefical too 🙂#2022-11-2610:49ikitommiwhat bad can happen if we add a mutable schema type? 🙈 https://github.com/metosin/malli/issues/786#issuecomment-1328020448#2022-11-2611:21pithyless> Just because we can, doesn't mean we should. 🙈 I feel if this was an issue posted on Clojure JIRA, it would require a lot more burden of proof on the poster to argue why this is really necessary, what are the practical use-cases that can't be solved otherwise, and whether the pros overcome all the cons (and future maintenance/compatibility). For example, is breaking the validate/validator referential transparency something we want to be doing so easily? Feels like a slippery slope... I don't have a dog in this fight, I don't really feel the pain of the original poster, but I'd argue prudence: maybe we should move this idea to an independent malli extension library until it shows its usefulness. Otherwise it will be hard to remove if it goes into malli-core and people depend on it. :)#2022-11-2613:10ikitommiI agree it should not be part of the core. I had a use-case in a project with custom reference types, where this might have been useful, not 100% sure. For the “hard to remove later”, there are some escape hatches https://github.com/metosin/malli#alpha: > • extender API: public vars, name starts with -, e.g. malli.core/-collection-schema. Not needed with basic use cases, might evolve during the alpha, follow https://github.com/metosin/malli/blob/master/CHANGELOG.md for details > • experimental: stuff in malli.experimental ns, code might change be moved under a separate support library, but you can always copy the old implementation to your project, so ok to use. maybe malli.experimental.mutable :thinking_face: (NOTE: just added the experimental documentation to README)#2022-11-2613:35ikitommioh, as the normal schemas capture forms eagerly, the whole parent-schema tree should be mutable to keep the child mutable. Not going to change all schemas to be lazy because of this.#2022-11-2613:36ikitommimutable = evil. thanks.#2022-11-2613:45ikitommi
(deftest schema-test
  (let [!schema (atom :int)
        mutable (mem/schema (fn [options] (m/schema @!schema options)))
        mutable-schema (mem/schema (fn [options] (m/schema [:map [:mutable mutable]] options)))
        immutable-schema (m/schema [:map [:mutable mutable]])]

    (testing "initial schema form"
      (is (= [:map [:mutable :int]]
             (m/form mutable-schema)
             (m/form immutable-schema))))

    (testing "mutating schema"
      (reset! !schema [:enum "so" "mutable"]))

    (testing "mutable schema changed ok"
      (is (= [:map [:mutable [:enum "so" "mutable"]]]
             (m/form mutable-schema))))

    (testing "immutable schema captured the orginal form"
      (is (= [:map [:mutable :int]]
             (m/form immutable-schema)))

      (testing "but workers see the mutated schema"
        (is (true? (m/validate immutable-schema {:mutable "so"}))))

      (testing "mutating schema again"
        (reset! !schema :boolean))

      (testing "form is still from the original value"
        (is (= [:map [:mutable :int]] (m/form immutable-schema))))

      (testing "validator is cached from the second value"
        (is (true? (m/validate immutable-schema {:mutable "so"}))))

      (testing "explainer is cached with the third value"
        (is (nil? (m/explain immutable-schema {:mutable true}))))

      (testing "conclusion"
        (is (not= "mutable" "good idea"))))))
#2022-11-2812:47jeroenvandijk@U055NJ5CC Thanks for trying to come up with a solution for #786. I’m reading your conclusion now 😅 Too bad#2022-11-2812:49jeroenvandijkI’ll create something internally and see how this goes. I do have use cases for this / multispec, but I can understand it might not be compatible with Malli’s design#2022-11-2813:01jeroenvandijkInspired by your example @U055NJ5CC, the following can work for me too
(defn multimethod-schema [mm]
  (let [dispatch (.dispatchFn mm)]
    (m/into-schema
     :multi
     {:dispatch dispatch}
     (map (fn [[type f]] [type (f {dispatch type})]) (methods mm)))))

(defmulti mm-example :Type)   
     
(def mm-schema 
  (if (= (System/getenv "ENV") "PRODUCTION")
    (constantly (multimethod-schema mm-example))
    (fn [] (multimethod-schema mm-example))))

 (defn valid? [x]
   (m/validate (mm-schema) x))

(valid? {:Type :x}) ;=> false

(defmethod mm-example :x [_]
  [:map [:Type [:= :x]]])

(valid? {:Type :x}) ;=> true
#2022-11-2813:02jeroenvandijkI don’t think I need much more than this, let’s see#2022-11-2813:02ikitommithat works, unless you wrap it in another schema, .e.g [:map [:mm (mm-schema)]]#2022-11-2813:03ikitommithere, :map eagerly creates the form from the current snapshot value.#2022-11-2813:04ikitommiI think it would be possible (and a good idea!) to be able to disable all caching in malli via an option, e.g. a dev-mode. would allow one to shoot him/her in the leg in new, imaginary ways. But, would also allow a “expected” repl experience, that could be turned of in normal/prod mode.#2022-11-2813:07ikitommiI think it would also clean up malli internals a lot if all caching would happen via the Cached protocol. swapping it’s impl would be trivial after that. so: 1. refactor malli-interanl caching to always use Cached 2. allow disabling caching via an option 3. welcome (optional) mutability for people who don’t have enough troubles already - or they know what they are doing 😎 #2022-11-2813:08ikitommiI can give it a shot at some point, interested in seeing if 1 would be a good idea.#2022-11-2813:11jeroenvandijkYeah interesting. To be honest I haven’t used all the features/optimizations of Malli yet. So I can’t judge how far such a change would impact things. I have been viewing Malli naively as an optimized, data oriented version of clojure.spec, but this of course has it’s own tradeoffs. I’m happy to burn myself with the above trick and report later how bad it was 😅#2022-11-2818:34jeroenvandijkBtw, I find the :multi construct very powerful. It’s also a nice tool to make the error messages very precise. I already adapted the multispec sugar to something custom to accomodate more precise errors. So maybe a generic multispec thing is not that useful after all. But I believe the option to disable caching for parts of the schema remains interesting. For now though performance is not my main concern#2022-11-2711:19CarloIs there a way to make :tuple accept lists, instead of vectors? What's the default way to spec a list?#2022-11-2711:56CarloI see that :cat takes a {:type :list} argument. I'm using this for the time being.#2022-11-2718:13ikitommiyes, :cat should accept all sequences, don’t think the :type hint does anything there. Would like to implement https://github.com/metosin/malli/issues/264 in near future, with that malli would understand the :type tags.#2022-11-2819:16mike_ananevHi, there. I’m little bit confused how to transform schema to real Swagger2 structure, from example https://github.com/metosin/malli#swagger2 ? I want to write message specs using Malli and generate some artefacts for other teams which use Golang or Java. I’m trying to figure out how to do this? I’ve implemented Malli Spec -> JSON Schema, but wanna use Swagger or Protobuf. The example above is incomplete and I don’t know how to generate real swagger file.#2022-11-2819:45valtteriIf you’re describing just data schemas, your swagger documentation would probably be something like
{
  "swagger": "2.0"
  "info": {"title": "My Title", "description": "something"},
  "definitions" {
    "MyObj1": <your malli->swagger-output-here>,
    "MyObj2": <your malli->swagger-output-here>,
  }
}
#2022-11-2819:46Ben SlessFor swagger you also need data like route and method which isn't part of the schema. Protobuf support doesn't exist yet afaik#2022-11-2819:47valtteriAFAIK swagger had slight (annoying) differences compared to JSON schema :thinking_face:#2022-11-2819:48valtteriBut anyways, as Ben said, Swagger doesn’t bring much (any) value if you’re just describing data. JSON schema should do the trick.#2022-11-2819:48valtteriThere are JSON schema libs for Java and Go#2022-11-2820:01mike_ananevOk, thank you. Got it! It would be good to transform Malli specs to Protobuf 3 schemas cause many teams use gRPC.#2022-11-2820:36Ben SlessTell them distributed objects are bad or wait until I get to PR it to malli? 🙃#2022-11-2916:07AkizHi, I use ‘Malli’ for ‘route data validation’. My problem is that when I validate a large structure and get an error, the output to Emacs and REPL is very slow (other IDEs don’t have this problem), I have to wait several minutes before I can work again (the error output has ~10000 lines). I was thinking of shortening the error somehow, but I have no clear idea how. Has somebody here solved a similar problem?#2022-11-2916:18respatializedmalli.error/humanize can reduce the error size, but I'm not sure if you've already tried that or if it will suffice for your problem.#2022-11-2916:18Noah Bogarthow are you performing the validation? could you do something like (assert (m/validate X Schema) (-> X (m/explain Schema) (me/humanize)))?#2022-11-2916:20respatializeda more general purpose solution might be to defer your error logging to an async logging library like https://github.com/BrunoBonacci/mulog, which provides a ring buffer mechanism that you can hook into with custom transformation functions to output errors and logs that narrow down to the specific thing you're trying to investigate.#2022-11-2916:21AkizI forgot to tell that humanization doesnt work for me.. for unknown reason, this is my coercion in reitit-hander. It would probably help a lot.
(r.coercion.malli/create (-> r.coercion.malli/default-options                                                                          (assoc :strip-extra-keys true)                                                                   (update :error-keys #(conj % :humanized))))
#2022-11-2916:24Akiz@UFTRLDZEW i use timbre currently, but i would like to solve that problem on malli / emacs level first.#2022-11-2916:34ikitommiseveral minutes 🙀#2022-11-2916:34AkizOh, the error is not related to reitit coercion. It is invalid return output value…#2022-11-2916:35Akiz@U055NJ5CC Yeah, vscode got it in second.#2022-11-2916:35ikitommido you have the pretty printing turned on?#2022-11-2916:36ikitommithe colored outputs can be slow, dunno if there is a bug in rendering, fipp is supposed to be fast.#2022-11-2916:36ikitommibut, not an emacs user (anymore), so not sure what happends there, if calva is fast with the same case.#2022-11-2916:41AkizDid you switch to vscode? Yeah, i do, i will try to disable it, thanks. Is there some easy way for instrumented fns to produce only humanized errors?#2022-11-2917:31Akiz• disabling pretty/reporter for instrumentation helped a lot.. closing other windows in emacs as well 😄. #2022-11-2916:58Ben SlessRegarding providing protobuf schema from malli, what do you think about defining a data representation for the protobuf spec (even as an external dependency), then defining the transformation from malli to it? I hate protobuf but it could be useful There are also limitations to account for, like oneOf can't be repeated, which make me pull my hair out#2022-11-3009:13dharriganI'm wondering, when invoking (mg/generate schema), if the schema contains something like [:enabled {:default true} :boolean], can the generate always use the default value, rather than flipping between true and false?#2022-11-3009:20jeroenvandijkI think you could do [:enabled {:gen/elements [true]}] Maybe I’m wrong, https://github.com/metosin/malli#value-generation#2022-11-3009:21dharriganoohh.. trying...#2022-11-3009:23dharriganYup, that works. ta! 🙂#2022-11-3020:00dharriganI'm a little confused as to where optional sits in a schema. It's not immediately clear in the documentation. Is this [:map [:name {:optional true} :string] same as [:map [:name [:string {:optional true}]]]?#2022-11-3020:18Chris O’DonnellI don't think so. Pretty sure :optional needs to be a parameter of the map key entry.#2022-11-3020:19Chris O’Donnell(The former)#2022-11-3020:19Noah Bogarti believe that the position of the {} is supposed to look like metadata on the following object#2022-11-3020:20Noah Bogartso [:map [:name :string]] and then with the optional "metadata"/params [:map [:name {:optional true} :string]]#2022-11-3020:22dharriganI see, so, [:map [:name {:optional true}]] means the name is optional. Then if I have [:map [:name {:optional true} [:string {:min 1 :max 4}]]], then it's not only optional, but I've also said that the string (if provided) must be between 1 and 4.#2022-11-3020:23dharriganso the metadata is about the proceeding "thing"#2022-11-3020:27Noah Bogartah yeah, it would be the previous thing not the following, as you've noted with the {:min 1} example#2022-11-3020:27Noah Bogart{:optional true} means that the key might not be present, whereas [:maybe X] means the value might not be present#2022-11-3020:29dharrigan👍#2022-12-0517:31escherizeThink of it like hiccup: [:div {:style "..red"} [:span ...]] the attributes maps apply to the tags immediately before them#2022-12-0517:50dharrigan👍#2022-12-0220:11Ben SlessHow about type variables for https://github.com/metosin/malli/issues/770?#2022-12-0416:02jfntnNot sure I understand, can you give an example?#2022-12-0416:17Ben Sless(map identity xs) is a classic example, identity doesn't have type any -> any, but a -> a, there's a clear relation between input and output#2022-12-0416:19jfntnThanks I see what you mean now, but isn’t that orthogonal to that particular issue?#2022-12-0417:02Ben SlessPartially. How will you write the schema for the higher order function map?#2022-12-0418:58jfntnRight, I think I see what you mean now. The instrumentation fn would unify the type variables with the actual values at runtime#2022-12-0222:56escherizein prismatic schema, you can use a class with s/defn. Is there a similar approach for mx/defn? I could check for instance in a [:fn] I suppose#2022-12-0407:25CarloA question about recursive generators and :+. Consider this recursive spec that describes a boolean formula:
(def formula
  [:schema
   {:registry
    {::formula
     [:or
      :boolean
      [:tuple [:enum :not] :boolean]
      [:tuple [:enum :and] [:* [:ref ::formula]]]
      [:tuple [:enum :or]  [:* [:ref ::formula]]]]}}
   [:ref ::formula]])
We can of course generate examples via:
(repeatedly 20 #(mg/generate formula))
But, if I change :* with :+ (rest in 🧵)
#2022-12-0407:29CarloI now have:
(def formula
  [:schema
   {:registry
    {::formula
     [:or
      :boolean
      [:tuple [:enum :not] :boolean]
      [:tuple [:enum :and] [:+ [:ref ::formula]]]
      [:tuple [:enum :or]  [:+ [:ref ::formula]]]]}}
   [:ref ::formula]])

(repeatedly 20 #(mg/generate formula))
and the generator for this seems broken to me (for starters, it usually doesn't return 20 things. I suspect the reason is that it makes more difficult to generate values (as a naive sampling terminates more rarely). Is that correct?
#2022-12-0410:31ikitommirecursive generators are hard, please read https://github.com/metosin/malli/blob/master/src/malli/generator.cljc#L186-L281. Ideas welcome on how to make it better.#2022-12-0602:14ambrosebsFYI I'm looking into it here: https://github.com/metosin/malli/pull/792#2022-12-0606:18ambrosebsthe issue is that recursive :+ can make generators like (gen/non-empty (gen/return ()))#2022-12-0606:43ambrosebsshould be fixed in https://github.com/metosin/malli/pull/792#2022-12-0607:20ikitommiThat was a quick resolution, thanks @U055XFK8V!#2022-12-0408:06CarloAlso, what's :malli.core/potentially-recursive-seqex supposed to represent? Can't seqexes contain recursive terms?#2022-12-0408:32CarloBy a brute force search, I saw that this problem is solved if I wrap :+ into a :schema. A more thorough explanation would be useful though#2022-12-0419:06jfntnCan someone confirm whether :schema values being explained as reified values is a bug?
malli.core
> (explain [:map [:a int?]] {:a "_"})
{:errors ({:in [:a],
           :path [:a],
           :schema #<
https://github.com/metosin/malli/issues/789
#2022-12-0505:36Ben SlessShould and generators use bind in reverse order to allow for transformers? Sort of like specs conforming#2022-12-0518:36dharriganI wonder if anyone may have an example of using {:gen/gen ....} to generate a random email address for a malli schema?#2022-12-0518:42pithylesson my phone, but are you familiar with https://github.com/gfredericks/test.chuck? I’ve previously used string-from-regex for this. I’ve also more recently been using the faker library with malli to generate reasonable looking data if you’re not testing the extremes of the data itself (and don’t need smart shrinking)#2022-12-0518:43Ben SlessI can whip one up if you'd like Tldr is I'd break it down to its components first, then compose their generators#2022-12-0518:45pithylesswith generators I think the question is always what’s “interesting” and in scope for the “purpose” of the output. If you’d like to test the limitations of the email string itself @UK0810AQ2 approach of composing from smaller generators is sound. #2022-12-0518:47dharriganActually#2022-12-0518:47dharriganI figured it out#2022-12-0518:47dharriganthre is an undocumented re-gen!!!#2022-12-0518:47dharriganIt pays to look at the source code#2022-12-0518:48dharrigan[:foo/email-address [:re {:gen/re-gen [email-pattern]} email-pattern]]#2022-12-0518:48Ben SlessYeah but the email regex is a monster#2022-12-0518:48dharriganYeah, it's hairy, been there before on validating email. I'll take a pragmatic approach.#2022-12-0519:50Noah Bogartis there any difference between [:id :int] and [:id int?]?#2022-12-0519:50Noah BogartI tried to search but it's kind of nebulous lol#2022-12-0519:55pithylessThey have implementation differences that leak into your code in surprising ways (if you're not aware of them). I suggest using :int by default. :) See e.g: https://github.com/metosin/malli/issues/636 https://github.com/metosin/malli/issues/327#2022-12-0519:59Noah Bogartah yeah, i remember seeing that a while back. that's annoying#2022-12-0616:22AbhinavThis was very interesting#2022-12-0619:05Noah BogartFound a difference between [:sequential {:max 1} schema] and [:? schema] when transformed to a json-schema: https://github.com/metosin/malli/issues/793#2022-12-0619:22Noah BogartLooks like none of the "malli.core/sequence-schemas" are handled by https://github.com/metosin/malli/blob/cda67420c215b0a8018e1adf70660826d1ba658f/src/malli/json_schema.cljc#2022-12-0621:34Noah BogartA related question: What edition of JSON Schema is Malli targeting?#2022-12-0621:16ambrosebsSome cleanup following the recent generator bugfix, including a function from schema -> code that generates generator. https://github.com/metosin/malli/pull/794#2022-12-0621:34Noah BogartA related question: What edition of JSON Schema is Malli targeting?#2022-12-1122:22Erik Colsonhi. I am trying out malli. What is the usual approach to validate a date string with malli?#2022-12-1203:29Ben SlessCoerce it first to the expected date type using a decoder#2022-12-1207:10dharriganHere's a little example that I have used (there may be other approaches)#2022-12-1207:11dharrigan
(def ^:private iso8601-message "is not in ISO DATE TIME format, e.g. 2020-07-03T10:15:30+01:00 or 2020-07-03T10:15:30Z")

;; ISO_OFFSET_DATE_TIME format, e.g., 2011-12-03T10:15:30+01:00 or 2011-12-03T10:15:30Z
(defn ^:private date-time-parser
  [date-time]
  (try
    (.parse DateTimeFormatter/ISO_OFFSET_DATE_TIME date-time)
    (catch DateTimeParseException _)))

(def date-range [:map
                 [:from {:optional true} [:fn {:swagger/type "string" :swagger/format "date-time" :error/message iso8601-message} date-time-parser]]
                 [:to {:optional true} [:fn {:swagger/type "string" :swagger/format "date-time" :error/message iso8601-message} date-time-parser]]])
#2022-12-1207:37Ben SlessIt's a bit draconian but I came to adopt the approach of never using a :fn schema if I can help it. Instead, all data coming into the process should go through a coercer. In reitit coercion is built in for you, otherwise just roll your own: https://clojurians.slack.com/archives/CLDK6MFMK/p1636145457067000?thread_ts=1636057867.062800&amp;cid=CLDK6MFMK Then just say your date is an inst?#2022-12-1207:41eskosI used to do what @U11EL3P9U does and it works well enough IMO. While not immediately apparent, I also created https://github.com/esuomi/muotti/ partly because of the complexities related to this problem, and in fact its source has a tidbit which may also be inspiration, if nothing else: https://github.com/esuomi/muotti/blob/d6e4e3df5f70bf1d26d035b034d9306445aea423/src/main/clj/muotti/malli.clj#L181-L199#2022-12-1208:05dharrigan👍#2022-12-1211:07Erik Colsonthanks for the examples. I am going to use @U11EL3P9U’s approach on this. @UK0810AQ2 what is the reason you prefer not to use :fn ?#2022-12-1211:16Ben Sless• fn schemas are opaque • values can't be generated • they hide the real type you're interested in. For example, if you have a time stamp, you might want to compare it against things (such as other dates), meaning you'll have to parse it anyway. You want the real type on your hands • Schemas specify ought, not is. you use decoding to make the best effort to bring what is to ought, then validate. You can often find an actual type which is closer to what you want than a fn#2022-12-1211:17Ben Slessalso see https://www.youtube.com/watch?v=IcgmSRJHu_8#2022-12-1211:19Ben SlessI can give you a real example - I have code which takes date over as string and needs to calculate N days ago from it, an API which returns numbers in JSON as strings. I don't want to do manual parsing. Once everything is past the gate of coercion and validation, I want my data to be the types I actually use, not what some random stressed out developer at $COMPANY crammed over a poorly thought out API#2022-12-1215:24Erik Colson@UK0810AQ2 That seems to make sense. Thanks for your explanation!#2022-12-1221:34DrLjótssonIs there a non-negligible performance penalty for sending in raw malli specs to validation functions instead of defining them with m/schema once and providing them to validation functions?#2022-12-1221:35DrLjótssonI.e.,
(def X [:int])
(m/validate x 9)
vs
(def X (m/schema [:int]))
(m/validate x 9)
#2022-12-1221:36DrLjótssonUnder the assumption that the validation function will be run multiple times of course.#2022-12-1305:25Ben SlessYou should build a validator if performance matters, can be two orders of magnitude faster#2022-12-1416:13ikitommioh, that’s a bug. fixing it in https://github.com/metosin/malli/pull/798#2022-12-1416:26ikitommimerged @US9EF3BGU#2022-12-1416:30rmxmstill I have another pickle... so when I run explain-data, I can obtain all error messages into a sec. however, when I define :fn it doesnt put error as "last" as mentioned before, I will paste the real code, the gist of the problem is that map with :error/message is at different index of the vector#2022-12-1416:48rmxm
(def schema
 [map
  [:x [:fn #(= % 2) {:error/message "not 2"}]]
  [:y [:string {:min 2 :error/message "longer than 2}]])

(def payload
 {:x 3
  :y "a"})

(->> (mu/explain-data schema payload)
     :errors
     (map #(-> % :schema last :error/message)))
#2022-12-1417:27ikitomminot sure what you are doing, but:
(def schema
  [:map
   [:x [:fn {:error/message "not 2"} #(= % 2)]]
   [:y [:string {:min 2 :error/message "longer than 2"}]]])

(def payload
  {:x 3
   :y "a"})

(->> (mu/explain-data schema payload)
     :errors
     (map #(-> % :schema m/properties :error/message)))
; => ("not 2" "longer than 2")
#2022-12-1417:27ikitommior you can use something like me/humanize:
(->> (m/explain schema payload)
     (me/humanize))
; => {:x ["not 2"], :y ["longer than 2"]}
#2022-12-1417:40rmxmgreat thanks... well I am trying to just aggregate error strings to push them as notifications... those examples work, thanks#2022-12-1419:25escherizeI want to decode a string with an enum like so: [:enum :A :B] + "A" => :A This doesn’t work:
(mc/decode [:enum :A :B] "A" (mtx/string-transformer))
;; => "A"
But all of these will work:
[(mc/decode [:and keyword? [:enum :A :B]]            "A" (mtx/json-transformer))
 (mc/decode [:and keyword? [:enum :A :B]]            "A" (mtx/string-transformer))
 (mc/decode [:enum {:decode/specific keyword} :A :B] "A" (mtx/transformer {:name :specific}))]
;;=> [:A :A :A] 
Is there another (even better?) way to do that?
#2022-12-1419:39Ben Slessdecode/string instead of specific?#2022-12-1419:53ikitomminot released, but https://github.com/metosin/malli/pull/782 last month 😎
(m/decode [:enum :A :B] "A" (mt/string-transformer))
;=> :A
#2022-12-1419:54escherizesorry @UK0810AQ2, I can’t parse that.#2022-12-1419:54escherize@U055NJ5CC Oh, neato#2022-12-1419:55escherizeIs that coming out in a release soon? (aka when?)#2022-12-1419:55ikitommiif you use deps, you can just point to the latest commit on master.#2022-12-1419:57ikitommiwill try to cut a release this year, need some work on the providers, a minor breaking change, so a minor bump to 0.10.0#2022-12-1419:57escherizeI think I’ll wait for that, then. Thanks for the info!#2022-12-1422:06escherizeIs there an analogous function to prismatic schema’s check? Here’s the docstring:
"Return nil if x matches schema; otherwise, returns a value that looks like the
   'bad' parts of x with ValidationErrors at the leaves describing the failures.

   If you will be checking many datums, it is much more efficient to create
   a 'checker' once and call it on each of them."
#2022-12-1422:06escherizehmm, is that a key in explain?#2022-12-1422:07escherizeI have this to work with:
{:schema [:map [:x int?]],
 :value {:y "3"},
 :errors ({:path [:x], :in [:x], :schema [:map [:x int?]], :value nil, :type :malli.core/missing-key})}
#2022-12-1422:08escherizeseems like I can reduce on errors, and assoc-in using :path or :in?#2022-12-1422:11escherizeanswering my own question here, using (me/humanize (m/errors $schema $value)) does similar thing to plumatic.schema/check #2022-12-1422:11escherizeanswering my own question here, using (me/humanize (m/errors $schema $value)) does similar thing to plumatic.schema/check #2022-12-1500:46rmxmanother quick question, similar to the last one, why?
(m/properties [:enum "x" "y" {:error/message "a"}]) ;; nil
(m/properties [:string {:error/message "a"}]) ;; #:error{:message "a"}
#2022-12-1506:12ikitommiproperties map should be the second element in the vector syntax, not the last.#2022-12-1518:05rmxmthanks a lot, yeah silly mistake... but more and more I get more confident I will be able to express any crazy schema + have validating/schema/errors in one shape, thanks for your help#2022-12-1501:54lepistaneI have a question. So when i am using spec to describe paramenters with reitit.ring.middleware.multipart/temp-file-part there is swagger/type which can be shown in the swagger page. I can't seem to find a way to do this with malli (yes i swapped coercion) I tried https://github.com/metosin/reitit/blob/master/modules/reitit-malli/src/reitit/ring/malli.cljc this breaks page so that you can't even put params anymore
:parameters {:query [:map {:json-schema {:type "file"}}
                                     [:filename string?]
                                     [:content-type string?]
                                     [:size int?]
                                     [:tempfile [:fn (partial instance? )]]]
this is the gist Is this solvable or i am stuck with spec? after some tryouts this worked
[:filename {:swagger/type "file"} string?] 
but i am wondering am i making a mistake and this should be done some other way? edit https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj provided all answers i needed, didn't see it first time i checked examples 🙂 thankful and grateful for the examples gratitude
#2022-12-1503:19lepistaneI guess i didn't find answer to all questions ... So i have custom schema that uses custom function for special case validation.
[:map {:closed true} 
 [:match-id [:fn #:error{:message "Invalid match-id"} #function[valid-uuid-as-str?]]] 
 [:date [:fn #:error{:message "Invalid date"} #function[valid-date-as-str?]]]]
But it doesn't work. I am already using this specialized validation in other places but i was thinking about reusing it all everywhere. is it possible that reitit parameter validation can be only primitive predicates (from clj.core and malli) ?
#2022-12-1506:29ikitommithere are no restrictions on reitit for using malli, it should just work. let me test that.#2022-12-1506:36ikitommitested and it just works :thinking_face:#2022-12-1506:37ikitommiif you can do a minimalist repro, I can check what’s wrong with it.#2022-12-1511:08lepistane@ikitommi Thanks! Here is the repo (just reused existing mali reitit example and added on top) https://github.com/StankovicMarko/reitit/blob/custom-validation/examples/ring-malli-swagger/src/example/server.clj You can see what i added on top: https://github.com/StankovicMarko/reitit/commit/8f6634d1f7529aebd23e6527ac738857894b0328 Main issue is that when i put values for GET plus, if they are wrong i can't make a request until i put correct value. picture A And for test route i can add whatever i want and i dont want that. I want to prevent it just like with 'native' types picture B#2022-12-1511:08lepistaneWhat am i doing wrong?#2022-12-1512:33lepistaneThis was the only i managed to do it
[:map
                            [:date {:description "Date when match happened"
                                    :json-schema/type "string"
                                    :json-schema/format "date"
                                    :json-schema/default "2022-10-10"}
                             [:and
                              [:re #"^\d{4}\-(0[1-9]|1[012])\-(0[1-9]|[12][0-9]|3[01])$"]
                              [:fn {:error/message "Invalid date"}
                               v/valid-date-as-str?]]]]
Use regex to enforce format of a string but feels wrong. shouldn't this already be available? https://swagger.io/docs/specification/data-models/data-types/#string
#2022-12-1512:34lepistaneAm i missing something?#2022-12-1513:50lepistane@ikitommi any ideas?#2022-12-1515:39Jordan RobinsonHello, I'm in the process of porting over a codebase from spec to malli, and it's very cool! Thanks for all your work. I have a small question that may be a bit stupid so forgive my ignorance, is there a way to pick up the seed or size values from within a :gen/fmap function?#2022-12-1516:44escherizeWhat I have done, is use a deterministic transformation on the generated value in :gen/fmap#2022-12-1516:47Jordan Robinsonthat sounds like it might be along the lines of what I'm looking for, I don't suppose you have a code snippet at all?#2022-12-1516:48escherizeits not a stupid question, let me see here#2022-12-1516:50escherize
(mg/generate
 [:and {:gen/fmap #(str "my_" % "_thing")} string?]
 {:seed 10, :size 10})
#2022-12-1516:50escherizesomething like that work for you?#2022-12-1516:55Jordan Robinsonah I see, I did try something like that at first but it didn't really work for my situation, let me try and write some pseudocode for what I have
(malli/schema [:fn
                 {:gen/fmap
                  (fn [_]
                    (generate-thing {:seed ???}))}
                 (fn [thing] (validate-thing thing))])
which I'm then calling via (mg/generate schemas/thing {:seed 0})
#2022-12-1516:55Jordan Robinsonso I'm not sure that would work here, does that make a bit more sense?
#2022-12-1516:57escherizecan you build up a generator, and supply it in the :gen/gen property?#2022-12-1516:57escherizemeaning a clojure.test.check.generators#2022-12-1516:58Jordan Robinsonpotentially yes, but this isn't actually in test code, so I'd really only rather use clojure.test.check.generators as a last resort since that would mean putting it in the src imports which, feels a bit funky#2022-12-1516:58Jordan Robinsonif it's not a common thing what I'm doing though I guess I'd have to go down that route#2022-12-1516:58escherizeI guess it depends how hard it is to generate your thing#2022-12-1516:59Jordan Robinsonnot hard but it does take an int seed#2022-12-1516:59Jordan Robinsonand I'm not sure how to pass that through#2022-12-1516:59Jordan Robinsonsince otherwise if I call it without the seed in tests, well, it's random 😅#2022-12-1516:59escherizeis it data? it might be worth using malli to describe it#2022-12-1517:00escherizethen you will get easier generation#2022-12-1517:01Jordan Robinsonthat's a good point, I guess I'm looking for an easy way out as we already have (generate-thing)#2022-12-1517:01escherizehow about…#2022-12-1517:01escherize
(mg/generate
 [:and {:gen/fmap #(generate-thing %)} int?]
 {:seed 10})
#2022-12-1517:02escherizeO_o#2022-12-1517:02escherizeprobably not super good, since the schema is meaningless now#2022-12-1517:02Jordan Robinsonaha I wasn't sure if that would work, as if I use that to validate as well will it try and think the input should also be an int?#2022-12-1517:02Jordan Robinsonyeah#2022-12-1517:02escherizeyou can use it to generate..#2022-12-1517:02escherize
(mg/generate
 [:and {:gen/fmap #(generate-thing %)} 
  [:or int? [:fn ..check for mything..]]
 {:seed 10})
#2022-12-1517:03Jordan Robinsonahh that's interesting#2022-12-1517:03escherizereally weird though#2022-12-1517:03Jordan Robinsonyes very weird, 😅 I guess that's how my brain went to trying to get a seed value in at first#2022-12-1517:03escherizeyeah not sure it’s super easy#2022-12-1517:04escherizeactually.. in:
(malli/schema [:fn
                 {:gen/fmap
                  (fn [_]
                    (generate-thing {:seed ???}))}
                 (fn [thing] (validate-thing thing))])
#2022-12-1517:04escherizewhat would be getting mapped over?#2022-12-1517:05Jordan Robinsonah nothing, but it does work, I ripped it from the malli test code#2022-12-1517:05Jordan Robinsonthis is the relevant example:
(is (= 42 (mg/generate [:re
                            {:gen/fmap (fn [_] 42)
                             :gen/schema :int}
                            #"abc"]))
#2022-12-1517:06Jordan Robinson(https://github.com/metosin/malli/blob/master/test/malli/generator_test.cljc#L171)#2022-12-1517:06escherizebut you can generate values from a regex#2022-12-1517:06escherizei guess you can generate values form a :fn schema too#2022-12-1517:08Jordan Robinsonthanks very much for your help btw, I'm about to finish today but you've given me some very interesting ideas#2022-12-1517:08escherizeNice! yeah I am starting on porting a giant codebase from spec and schema (with plenty of custom macros) to malli. Fun times!#2022-12-1517:09Jordan RobinsonI'm in somewhat of a similar situation, it's fun till it isn't has been my experience so far 😅#2022-12-1517:10escherizehope your pulse doesn’t rise too much over it 😛#2022-12-2211:15Jordan Robinsonas an update on this, I ended up writing a custom generator using the clojure test check generators in the end, thanks again for all your help#2022-12-1516:38lepistaneI am unable to find a solution to what should be simple thing. How do i convert date string to LocalDateTime object automatically? (i am assuming this is request coercion) This is my muuntaja instance
(def muuntaja-instance
  (m/create
   (-> m/default-options
      (assoc-in [:formats "application/json" :decoder-opts] {:decode-key-fn csk/->kebab-case-keyword
                                                             <<???>> #(java.time.LocalDate/parse %)})
      (assoc-in [:formats "application/json" :encoder-opts] {:encode-key-fn csk/->camelCaseString
                                                             :date-format "yyyy-MM-dd"}) 
#2022-12-1517:10Ben SlessYou have to define your own schema type for that, unfortunately, but it's pretty easy#2022-12-1517:11escherizeyesterday I wrote an x/defn macro that behaves like Schema’s s/defn. (mx/defn does not throw with incorrect :in or :out.) Is there a library or project to mirror little utilities like that in Schema or Spec?#2022-12-1517:11escherizeWhat other migration strats have you found useful?#2022-12-1517:58dvingothere is also https://github.com/CrypticButter/snoop#2022-12-1517:59dvingomx/defn should throw when instrumentation is enabled#2022-12-1517:59escherizeit does#2022-12-1518:00escherizesnoop is cool, but i want to stick to vanilla ways of doing things#2022-12-1518:00escherizemy macro just calls instrument at defn creation time#2022-12-1518:01escherizewith a specific filter to match only the current function#2022-12-1518:09ikitommiwanted to keep mx/defn slim. thought of adding the Schema-style :always|never-validate hints too.#2022-12-1519:17escherizehow would you implement the never validate mode?#2022-12-1519:18escherizeI have a macro emitting the function with an extra {::validate! id} metadata and emitting an instrumentation filter after doing the core/defn call. so that’s a way to do the always-validate behavior#2022-12-1519:18escherizenot sure how to guard it, maybe add something into mi/-strument where it filters out on a certain fxn metadata value#2022-12-1600:06escherizerelevant bit of :always-validate https://github.com/metabase/metabase/pull/27218/files#diff-e29756aed1e2e2cbce8a26a8e4b2e1cbedfb83cd4d853cf6942455e49516e4d8R44#2022-12-1518:00Ben Sless@ikitommi taking the coercer discussion here to thread#2022-12-1518:00Ben SlessI assume the 1 arity for coercer decodes or throws, would be nice if we could have 3 arity for on success and on failure continuations, where the on-success takes the decoded value and on-failure takes the explained result#2022-12-1518:10ikitommisure, that would work as the arities woudn’t clash.#2022-12-1518:11ikitommiwant to make a PR?#2022-12-1518:12ikitommiI like the idea. control to the user.#2022-12-1518:33Ben SlessWill do#2022-12-1518:35Ben SlessSketch for more robust time schemas#2022-12-1518:35Ben Sless@ikitommi feedback / ideas before I pursue this path further?#2022-12-1518:36Ben SlessAnd thoughts regarding adding an epoch transformer?#2022-12-1606:10ikitommiLooks good. Quick comments: • cljs + bundle size. If you want to use malli.time, it will brings the generators in, as multimethods don't get DCEd, it's a lot of extra. README defines how to check the bundle size reports. Option to have malli.time and malli.time.generators , which sucks too. • If this is only for Clj now, then not a big issue. Could be then malli.experimental.time - when temporal/cljs solution comes, it most likely changes a bit. • Schema types could be namespaced with time, e.g. :time/local-date, thinking of doing same for malli.util schemas.#2022-12-1607:33Ben Sless• cljs: not really sure how to approach that so I just punted and did clj only • will move to experimental for now • namespacing: I thought I did#2022-12-1609:01ikitomminamespace, so you did, didn’t notice, brilliant! 🙂#2022-12-1609:14juhoteperiI think it would be OK to merge and maybe even release clj-only version first#2022-12-1609:14juhoteperiThough there is some danger in that case that something in the API will be hard to implement on Cljs side#2022-12-1609:18juhoteperiI could explore using date-io as a facade to different JS date libs, so users can use whatever they want: https://github.com/dmtrKovalenko/date-io#projects#2022-12-1609:33juhoteperiAdding clj-only as experimental ns sounds good to me#2022-12-1615:28valtteriPersonally I’m a bit concerned about the longevity of date-io and would appreciate malli.time to be built upon something that will last over time. However I trust @U061V0GG2 ‘s opinion over mine 🙂 #2022-12-1615:29juhoteperiYeah the issue about Temporal support on date-io shows the the API isn't very well designed#2022-12-1615:30juhoteperiBut building our own thing to to support different libs isn't much better either#2022-12-1805:41Ben SlessBesides separating out generators code and providing acceptors for json schema, any other things to keep in mind?#2022-12-1811:53Ben SlessDraft MR is ready, your feedback is appreciated https://github.com/metosin/malli/pull/802#2022-12-1905:57Ben Sless@ikitommi I think it's ready for review, besides adding tests the design is pretty settled. Only thing I'm not sure about is the parsing code which exists in the core ns.#2022-12-1520:50escherizeBefore I build it myself — Is there a way to get a descirption of a schema in malli.core? e.g. [;map [:x int?]] => “A map containing: x, an integer”#2022-12-1522:47escherizeIf it did, Given a malli schema, it should return a string with a description of the shape it expects.#2022-12-1522:48escherizeAre there problems with doing that? I figure a call to m/ast, then dispatching on :type should be sufficient. wdyt?#2022-12-1605:48ikitommiyou should use m/walk + dispatch on type. Each Schema implements it's own -walk so all the children are walked correctly, see https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc on how to do this.#2022-12-1605:49ikitommibut, an interesting idea!#2022-12-1616:12escherizeCould you help me understand the benefits of using walk vs ast? Is walk more stable?#2022-12-1616:37ikitommiAll schemas implement the m/-walk. So you can walk over any schema, even those defined in user space. (m/walk schema (m/schema-walker identity)) works always. With AST, you have to know all schemas ahead of time to know how to handle those. In your use case, I guess you need to handle all schemas anyway, you can do the same with AST. Just that it breaks if someone changes the AST for a schema (each schema controls how it looks). There is a disclaimer in Malli README not to use the AST as a persistence model, could change, no no plans to break it and will avoid doing that. We are using AST walking too in projects. But, walk is always safe.#2022-12-1616:38escherizeThanks for explaining that 🙏 I am looking into walk and schema-walker now.#2022-12-1617:05escherizeHere’s a self contained example. I used cond since its easier to fit. I’ll actually use a multimethod though
(mc/walk
 [:map [:x :int] [:y :string]]
 (fn [schema _path children _options]
   (println "-------")
   (println "schema:" schema)
   (println "children:" children)
   (cond
     (= (mc/type schema) :int) "INTEGER"
     (= (mc/type schema) :string) "STRING"
     (= (mc/type schema) :map) (str "map of "
                                    (str/join "" (mapv (fn [[k _ v]] (pr-str [k v]))
                                                       children)))
     :else schema)))
#2022-12-1617:05escherizeI don’t think schema-walker is the way to go, since I dont want to ultimately return a schema#2022-12-1617:09escherizemaybe I can attach the stringified explaination to the schema as a property, and look for the prop on children?#2022-12-1617:09ikitommiI would copy the malli.json-schema and start from that#2022-12-1617:10ikitommiit's a postwalk, you get the transformed children to your callback, should be straightforward to implement what you want#2022-12-1606:10ikitommiLooks good. Quick comments: • cljs + bundle size. If you want to use malli.time, it will brings the generators in, as multimethods don't get DCEd, it's a lot of extra. README defines how to check the bundle size reports. Option to have malli.time and malli.time.generators , which sucks too. • If this is only for Clj now, then not a big issue. Could be then malli.experimental.time - when temporal/cljs solution comes, it most likely changes a bit. • Schema types could be namespaced with time, e.g. :time/local-date, thinking of doing same for malli.util schemas.#2022-12-1613:35Thomas Moermansmall q: I sometimes see ?something arguments with the ? prefix in e.g. in Malli and other metosin code bases, is that a convention for a "maybe" (x or nil) argument?#2022-12-1613:50Ben SlessYes#2022-12-1613:50Thomas Moermancheers#2022-12-1615:44ambrosebsFWIW when I see ?schema I think "coercable to a Schema".#2022-12-1615:45ambrosebsand schema as more like "an instance of Schema"#2022-12-1615:45ambrosebsat least, that helped with my reading of malli.core#2022-12-1615:51Ben SlessOh yeah It's a rougher sense of Maybe Could be nil, like ?properties, could be "could be a schema, don't know yet"#2022-12-1617:08Thomas MoermanAh, yes, i see.#2022-12-1617:10Thomas MoermanThe coercion makes sense as well, especially in the context of Malli.#2022-12-1815:54respatializedbasic question: is there a mechanism for annotating/adding descriptions to basic schema types in a manner similar to annotating map keys? Example use case: annotating tuple elements. I'd like to attach more information than just :string alone. E.g. instead of:
[:tuple :uuid :string :string]
something like:
[:tuple {:type :uuid :description "The unique identifier for thing."} {:type :string :description "The thing's annotation."} {:type :string :description "The thing's title."}]
is there an obvious way to do this that I've just totally overlooked?
#2022-12-1816:08ikitommiAny schema can have properties, so: `[:tuple [:uuid {:description "..."}] [:string {...}] [:string {...}]]`#2022-12-1816:10respatializedyeah I remembered the [:string {:description "..."}] syntax not long after I posted this, thanks! duckie#2022-12-2222:21Tiago Dall'Ocaquick question: is it normal for (mi/collect!) to be slow? I thought this macro could be at the end of every ns using defn schema metadata, but wouldn't be practical if it slowed down ns file evaluation 😕#2022-12-2222:48Tiago Dall'Ocaactually putting (do (mi/collect!) nil)) at the end of the file solved the issue#2022-12-2222:49Tiago Dall'Ocathe problem was emacs trying to print a whole bunch of text (apparently more then 1MB!)#2022-12-2222:48Tiago Dall'Ocaactually putting (do (mi/collect!) nil)) at the end of the file solved the issue#2022-12-2311:36armedHi, is there a way to apply malli function schemas (`m/=>`) to macros? Minimal example:
(def ?ErrOrOpts
  [:or
   [:fn {:error/message "should be Throwable"} throwable?]
   [:map {:error/message "should be a map"}
    [:throwable [:fn {:error/message "should be Throwable"} throwable?]]]])

(def ?LogArgs
  [:function
   [:=> [:cat ?Message] :nil]
   [:=> [:cat ?Message ?ErrOrOpts] :nil]])

(m/=> foo ?LogArgs)
(defmacro foo
  ([msg] nil)
  ([msg err-or-opts] nil))

(comment 
  (foo "hello")
  )
blows up with:
Invalid function arity (3):

  ((foo "hello") nil "hello")
....
It seems that macro call is wrapped
#2022-12-2311:44armedthe workaround seems is prepend args with two :any, e.g. [:=> [:cat :any :any <real marco args>] :nil]#2022-12-2311:45armedbut error messages will contain all that extra stuff#2022-12-2313:46armedseems it is not possible to instrument macros properly, better to instrument functions which macro calls#2022-12-2706:13orestisI see there’s some datetime related work, so perhaps I just need to be patient - I noticed that inst? schemas generate java.util.Date whereas I need java.time.Instant. I can definitely provide my own generator or even converter, but I was wondering if I’m missing something. #2022-12-2706:14Ben SlessThat generator will be in my MR#2022-12-2706:15Ben SlessI'll probably get back to working on it today#2022-12-2706:15Ben SlessAnd plz review! Feedback needed#2022-12-2711:44Ben SlessAaaand the experimental time schemas MR is ready 😎#2022-12-2806:15Jungwoo KimI’m searching for datetime stuff today and meeting this thread and looked up your MR. Thank you for great work.#2022-12-2711:44Ben SlessAaaand the experimental time schemas MR is ready 😎#2022-12-2823:17ThomasCWhat’s the best way use properties with malli.core/-simple-schema? Here’s an example:
(-> (m/-simple-schema {:type :=2
                       :pred #{2}
                       :properties {:here? 1}
                       :type-properties {:or-here? 2}})
      m/properties)
=> nil
I would have expected the map from either :properties or :type-properties to be returned from malli.core/properties
#2022-12-2905:52Ben SlessProperties go on an instance, type properties go on a simple schema definition The constructor could be documented better, so unless you want to read the source, what are you trying to solve?#2022-12-2910:58ikitommiLike Ben said: • :type-properties are shared for all instances, and do not contribute to form:properties are defined into the instance, e.g. [my-type {:kikka "kukka"}] • there are no utilities for reading both (could be), currently the schema applications (like json-schema transformation) merges those when reading, see https://github.com/metosin/malli/blob/master/src/malli/json_schema.cljc#L172#2022-12-3007:45Lucy WangHello, how should I achieve "use default value for a field when validation fails and the value is not compatible with the schema "? e.g.
(def MySchema
  [:map
   [:some-int-field {:default 30} :int]
   [:some-boolean-field {:default false} :boolean]])

(m/decode
  MySchema
  {:some-int-field "typo" :some-boolean-field "another-typo"}
  what-transformer-to-put-here--i-dont-know?)
The builtin default-value-transformer would do nothing when the field is present but invalid.
#2022-12-3011:11ikitommiGood Question! There is no such thing atm but doable with the current transformers: • on :leave, validate the value, if not valid, replace with "the default". You can access the schemas (and the properties) in the transformation using the :compile hook code would be 90% identical to the default-value-transformer , maybe could be just an option to it :thinking_face:#2022-12-3011:13ikitommigood case to try to learn how the transforformations work. Want to give it a try?#2022-12-3102:53Lucy WangAwesome! I actually ended with something exactly the same as you suggested (got some inspiration from https://github.com/metosin/malli/issues/143 )
(def fix-with-default-transformer
  (mt/transformer
    {:name :fix-with-default
     :default-decoder
     {:compile
      (fn [schema _]
        (if-let [{:keys [default]} (m/properties schema)]
          (fn [x]
            (if (m/validate schema x)
              x
              default))
          identity))}}))

  (def my-transformers (mt/transformer
                       mt/string-transformer
                       fix-with-default-transformer))
  (m/decode [:int {:default 100}] "99" my-transformers)
  ;; => 99
  (m/decode [:int {:default 100}] "what a typo!" my-transformers)
  ;; => 100
#2022-12-3103:00Lucy WangOne more thing I want is to be able to log the map field name that has this invalid value detected and replaced, e.g.
[:map
   [:number-of-adults {:default 1} [:int {:min 1}]]
   [:number-of-babies {:default 0} [:int {:min 0}]]]
and when I receive invalid input {:number-of-adults "foo"} I'd like to log this down
[WARN] Got invalid input for :number-of-adults "foo", use default value '1' instead
But as I search around it looks like the path information is not available to the transformers. So I think the only way to achieve that is to use m/walk to walk the schema and add the field name to the properties, e.g. the above schema would be enriched to be:
[:map
   [:number-of-adults {:default 1 :report/name :number-of-adults} [:int {:min 1}]]
   [:number-of-babies {:default 0 :report/name :number-of-babies} [:int {:min 0}]]]
This shall work, albeit being a bit verbose
#2022-12-3108:07ikitommiLooks good. Quick comments: • it’s better to apply the transformation on :leave so it’s postwalk. If you just return a function from :compile, it maps to :enter phase, so it’s a prewalk. If you use nested defaults, the top-level might fail on enter, as the children are not transformed. • returning nil from :compile is better than identity -> the transformation engine knows “there is nothing to do” • you can create the (pure) validator just once, much faster • so:
(defn fix-with-default-transformer []
  (mt/transformer
   {:name :fix-with-default
    :default-decoder
    {:compile
     (fn [schema _]
       (if-let [{:keys [default]} (m/properties schema)]
         (let [valid? (m/validator schema)]
           {:leave (fn [x] (if (valid? x) x default))})))}}))
• for the paths… what would you expect as a warning path from this:
(m/decode 
 [:map 
  [:x [:map 
       [:y [:int {:default 100}]]]]] 
 {:x {:y "what a typo!"}} 
 my-transformers)
;; => {:x {:y 100}}
#2022-12-3113:31Lucy WangThanks! The validator tip is very useful. > for the paths… what would you expect as a warning path from this: I'll expect something like [:x :y] would be perfect.#2022-12-3017:11escherizeI might have missed this, but how do you pronounce malli? 🙂#2022-12-3017:20ikitommiGoogle pronounces it bit lazily, but close enough https://translate.google.fi/?hl=fi&amp;sl=fi&amp;tl=en&amp;text=malli%0A&amp;op=translate#2022-12-3017:22ikitommi... and, will check the describe-PR tomorrow#2022-12-3019:34escherizeNo rush, I have a copy pasted namespace in the project now. But looking forward to a version bump for a few reasons 🙂#2022-12-3108:34ikitommiThis is really good. Merged.
(is (= "one of <:dog = a map where {:x -> <integer>} | :cat = anything> dispatched by the type of animal"
         (med/describe [:multi {:dispatch :type
                                :dispatch-description "the type of animal"}
                    [:dog [:map [:x :int]]]
                    [:cat :any]])))

  (is (= "one of <:dog = a map where {:x -> <integer>} | :cat = anything> dispatched by :type"
         (med/describe [:multi {:dispatch :type}
                    [:dog [:map [:x :int]]]
                    [:cat :any]])))
#2023-01-0400:00escherize🙏#2023-01-0400:00escherizehappy new year 🙂#2023-01-0408:51ilmoIPA: /ˈmɑ.lːi/#2022-12-3114:37prncHello 🙂 Just checkin’—there is a coercion section in the docs https://github.com/metosin/malli#coercion but it’s something that’s not available in the latest build available on clojars https://clojars.org/metosin/malli (`0.9.2`), right?#2022-12-3117:13ikitommiNot released yet.#2023-01-0212:53Ben Slesswoohoo, time schemas merged partyparrot#2023-01-0213:40robert-stuttafordthank you!#2023-01-0711:24Jungwoo KimThank you!#2023-01-0309:11ikitommiwould this a good or a bad change? https://github.com/metosin/malli/pull/809, comments welcome.#2023-01-0313:15Ben SlessThe change is good, but the intent is no longer clear from the function's name#2023-01-0407:53ikitommiFully agree. Also, sometimes less is more. Not going to merge that.#2023-01-0407:55Ben SlessMaybe more convenient would be a variadic arity into-registry function that takes N arguments and tries to make a registry out of them#2023-01-0407:56ikitommilike composite-registy -> into-registry?#2023-01-0408:08Ben Sless🙂#2023-01-0408:08Ben Slessmaybe expose it from malli.core?#2023-01-0418:42escherizeWill me/with-spell-check ignore/passthrough if I use it on open maps or schemas? Or should I add a check to see if maps are closed before using with-spell-check?#2023-01-0513:15ikitommispell checking works on m/explain results. So, it will not see extra keys. You can close the maps with malli.util/closed-schema before calling it.#2023-01-0513:15ikitommiGood idea might be to add levels to explain, e.g. emit :warning for extra keys on open maps. then it would work for all maps.#2023-01-0513:30ikitommionly closed maps emit the "extra key" errors, which the spell checker reads, so in that sense, does not work with open maps.#2023-01-0516:57escherizeActually that’s the behavior I was hoping for. Thanks @U055NJ5CC#2023-01-0419:06escherizeAlso, just a note with malli.experimental.describe — should we find a way to unify it with :error/message or :fn/error? Right now I have a utility to assoc the same string into :describe and :error/message props, but it feels very w.e.t. (not d.r.y. + write everything twice)#2023-01-0513:17ikitommiUnification is a good idea. Issue + ideas on implementation on it most welcome.#2023-01-0513:18ikitommirelated, will work on type-tagging soon, will help to resolve these:
(defmethod accept 'float? [_ _ _ _] "float")
(defmethod accept :float [_ _ _ _] "float")
#2023-01-0608:35ikitommi@escherize does this look correct? https://github.com/metosin/malli/pull/813/commits/55975911a2bb653c10063968b33bcf35f1d21908#2023-01-0802:40DFSTThis might be a basic question, but I'm a little confused about what the numbers signify in a given path that's returned by subschemas . In the example from the readme, for example, one path that's returned is [0 :address 0 :lonlat 1]. What are the numbers for in the vec? Is there documentation somewhere that explains how to interpret a path that I missed?#2023-01-0804:37Ben SlessYou have some schema there that models an alternative. Either alt or or?#2023-01-0808:19ikitommiyes, each schema always accumulates to the :path (path in schema) even if they do not contribute to the :in (path in value). Each Schema defines it’s own meaning for :path, but by convention, numbers represent child indexes, starting from 0. e.g. • [:tuple :int :int] has indexes 0 & 1 • [:maybe :int] has just index 0#2023-01-0808:19ikitommi
(def schema [:map [:x [:tuple :int [:maybe [:map [:y [:and :int [:> 6]]]]]]]])

(mu/subschemas schemas)
;[{:path [], :in [], :schema [:map [:x [:tuple :int [:maybe [:map [:y [:and :int [:> 6]]]]]]]]}
; {:path [:x], :in [:x], :schema [:tuple :int [:maybe [:map [:y [:and :int [:> 6]]]]]]}
; {:path [:x 0], :in [:x 0], :schema :int}
; {:path [:x 1], :in [:x 1], :schema [:maybe [:map [:y [:and :int [:> 6]]]]]}
; {:path [:x 1 0], :in [:x 1], :schema [:map [:y [:and :int [:> 6]]]]}
; {:path [:x 1 0 :y], :in [:x 1 :y], :schema [:and :int [:> 6]]}
; {:path [:x 1 0 :y 0], :in [:x 1 :y], :schema :int}
; {:path [:x 1 0 :y 1], :in [:x 1 :y], :schema [:> 6]}]

(mu/get-in schema [:x 1 0 :y])
; [:and :int [:> 6]]

(mu/get-in *1 [0])
; :int
#2023-01-0808:21ikitommithere are also utilities for going from between the two, mu/path->in and mu/in->paths (could be many)#2023-01-0815:43DFSTThanks! I'm trying to piece together something to derive different subscription and event paths for a re-frame db given my database schema. Given my pretty basic skillset, this is a bit of a stretch goal for me, but this is really helpful!#2023-01-0908:14ikitommi:in is the path in the value (e.g. the db), did you try that?#2023-01-1015:27ikitommiRelated to https://twitter.com/cljtogether/status/1606883291982102529, I will be spending more time on OS this year, #malli and #reitit being the first targets. We have also planned (at :metosin:) better ways to organise around our OS development, a blog post on that soon.#2023-01-1118:28escherizeThis is wonderful#2023-01-1015:52Noah Bogartthat's great news#2023-01-1113:39CarloI have a question on the workflow - I need to check my understanding; ideally I would just put a (dev/start! {:report (pretty/reporter)}) inside my user.clj so that it's executed at startup time. But, if I understand correctly that only start instrumentation for the currently loaded modules (of which there's none at the time of user.clj execution). A workflow that works is loading some ns, then go back and dev/start! again, but is there a workaround that also keeps track of the loaded modules?#2023-01-1114:03ikitommiAlso interested in better ways to do this. You could hook it into integrant/etc workflow or make a shortcut for running that. Best would be if the would follow the new Var registration itself, tried that once, but got dirty quick. Ideas most welcome.#2023-01-1115:02CarloThank you, I agree that morally this should belong to a hook on var interning, but I did not know that was possbile at all. What would be the entry point for that?#2023-01-1116:50pithylessMy understanding was that would instrument new namespaces correctly, assuming that one used m/=> and not eg. via metadata on defn.#2023-01-1117:43Carlo@U05476190 there doesn't seem to be a difference between m/=> and metadata for this, do you have an example in mind?#2023-01-1115:16Noah BogartI'm struggling to get decoding working correctly. I want to transform a string timestamp to a clj-time instant: #clj-time/date-time "2023-01-11T15:14:48.964Z". (Yes, I know clj-time is deprecated lol). Do I have to define a custom schema for this?#2023-01-1118:31Ben SlessYes You can look at how the time schemas are implemented#2023-01-1118:32Noah BogartOh, did those get merged? I remember seeing your PR but didn't follow closely. They've not been released yet, right?#2023-01-1118:33Ben Slessmerged but not released#2023-01-1118:33Ben Slessif you're on deps you can enjoy them right now#2023-01-1118:34Noah BogartSadly, we're using leiningen 😭#2023-01-1118:34Noah BogartI'll look at the code on master tho, thanks#2023-01-1118:30escherizeAm I doing something wrong here? I thought these would be using the same schema underneath. Maybe I was mistaken
[(m/validate [:double {:min 1}] -1.0)
 (m/validate [double? {:min 1}] -1.0)]
;; => [false true]
#2023-01-1118:51ikitommipredicate schemas double? don’t read any properties like real schemas :double. As this is a common cause of confusion, options to fix: 1. drop support for predicate schemas, kinda extra anyways 2. map predicate schemas into real schemas 3. finish “schema of schemas” to describe allowed properties so the latter would give a warning in… dev-mode #2023-01-1119:28escherizeYou mentioned something about not needed to define a describe multimethod for both 'integer? :integer. Is that related to 2.? It seems better to have a single schema for both. I can’t see any downsides.#2023-01-1119:29escherizeI think the only way it would break is if you had [double? {:min 1}], and it was being ignored. Which a) never happened b) is wrong anyway.#2023-01-1120:13ikitommi1️⃣ :registry property
[:schema {:registry {"ConsCell" [:maybe [:tuple :int [:ref "ConsCell"]]]}}
 "ConsCell"]
2️⃣ :let schema
[:let ["ConsCell" [:maybe [:tuple :int [:ref "ConsCell"]]]]
 "ConsCell"]
#2023-01-1218:26escherize:let is clearer, But :registry has the benefit of restricting shadowing / duplicate schema refs#2023-01-1209:16Lucy WangFYI, looks like metabase starts to use malli instrumentation to replace plumatic schema based instrumentation https://github.com/metabase/metabase/pull/27303/files#diff-e29756aed1e2e2cbce8a26a8e4b2e1cbedfb83cd4d853cf6942455e49516e4d8R70-R72#2023-01-1217:46escherizeThanks for noticing! 🥲 — using it to replace some clojure spec things too.#2023-01-1217:22ikitommi🥳 thank you all. A good release!#2023-01-1220:29Dallas SurewoodAnyway to make the default of a spec property to be an adjacent key? As in the default of temp-hp is max-hp#2023-01-1221:18escherizeCan you fill in what you mean a little more?#2023-01-1221:18escherizeyou probably read this already: https://github.com/metosin/malli#default-values#2023-01-1221:18Dallas SurewoodI did#2023-01-1221:21Dallas SurewoodI was hoping to use m/decode to set the defaults of a map conforming to a spec. But all of the examples show default as a constant (and the ability to use default-fn). I was hoping to map a relationship between one property and another so if that property was missing, it's default would be another property. For instance, if temp-hp is not present, it should be set to whatever max-hp is#2023-01-1221:22Dallas SurewoodBut this is difficult to pull off.#2023-01-1221:25escherizeCan you lookup max-hp inside a function?#2023-01-1221:25escherizeor, how is it passed in?#2023-01-1221:29escherizedoes this solve it
(mc/decode [:map {:decode/player-init
                  (fn [{:keys [max-hp temp-hp] :as player}]
                    (if-not temp-hp
                      (assoc player :temp-hp max-hp)
                      player))}
            [:max-hp int?]
            [:temp-hp int?]]
           {:max-hp 50}
           (mtx/transformer {:name :player-init}))
;; => {:max-hp 50, :temp-hp 50}
#2023-01-1221:31Dallas SurewoodI think that does it but I'm just looking at solutions and seeing if that's reasonable enough to write over and over. Deciding how far I can push Malli to define the shape of my maps#2023-01-1221:31Dallas SurewoodAnd whether I can make it do these things somewhat automatically#2023-01-1221:34escherizeI might write a function to generate the schema#2023-01-1221:35escherizeNot sure I can help without more info#2023-01-1221:35Dallas SurewoodI think I'm moving on from it, but thank you#2023-01-1220:30Michael Gardneris there a recommended way to wrap spec.alpha specs in Malli schemas? Of course I can do:
[:fn #(s/valid? ::foo %)]
...but then the error message is unhelpful. I can fix that too, but felt like I might be duplicating someone else's effort.
#2023-01-1300:43escherizeIs anyone setting up mock functions with malli? It’s so cool!
(mx/defn eff :- string? [x :- int?]
  (str x))

(def mock-eff (mg/generate (m/form (:schema (meta #'eff)))))

;; mock called with good input, returns output that matches the return schema
(mock-eff 3)
"72eSV3WQpD6d9"

;; mock throws (just like the real thing would) on bad input
(try (mock-eff "3")
     (catch Exception e (ex-data e)))
;; => {:type :malli.core/invalid-input,
;;     :message :malli.core/invalid-input,
;;     :data {:input [:cat int?], :args ["3"], :schema [:=> [:cat int?] string?]}}
#2023-01-1307:08jeroenvandijkNice trick! Thanks for sharing. Maybe also interesting to start with this implementation and complete it when you need to.#2023-01-1309:46ikitommi@escherize Malli instrumentation has a :gen option for the lazy:#2023-01-1315:29pithylessHow would you approach customizing the generators for these kind of mocks? Would it be custom :gen at the mx/defn level? Or override the schema with custom registry at the (partial mg/generate)? Or something else?#2023-01-1315:51ikitomminot at the computer, but I think you can add :malli/gen meta to the var#2023-01-1316:25escherizeI generally have dev running when.. in.. dev. Is there a filter for that gen key?#2023-01-1316:26escherizeIt’d be nice to put a metadata on a function definition and have it autogen, then take it off recompile and no autogen. Maybe this already exists#2023-01-1314:27jeroenvandijkSo in theory you could build an application this way without filling in the details and have it somewhat working. And if it doesn’t work it would be logical errors like reusing a password, maybe other uniqueness like things, but you could get pretty far I guess. Sounds good!#2023-01-1315:24escherizeI think it’d be good for mocking an api for your hypothetical front end team#2023-01-1319:16Noah BogartWould there be any interest in having humanize say the type that's given? Instead of just {:some-num ["should be an integer"]}, it could say {:some-num ["should be an integer, given a string"]}#2023-01-1323:07escherizeyou can get something similar with https://github.com/metosin/malli/blob/master/docs/tips.md#getting-error-values-into-humanized-result#2023-01-1323:08escherizebut I agree it’s better to say the type that was given#2023-01-1408:56ikitommiyes, adding the humanized value type into the error message makes sense - as an optional :wrap impl for example.#2023-01-1319:30Dallas SurewoodWhen writing encoders/decoders for a transformer, is there a way to match by schema and not just :int or function symbols? For instance
(defn x-transformer [] (mt/transformer
                     {:decoders {'str-stat-schema (fn [x] (:value x))}
                     :encoders {'str-stat-schema (fn [x] {:value x})}}))
Was hoping that would work. Would like to make decoders to convert one schema to another
#2023-01-1408:57ikitommiwhat is a 'str-stat-schema here?#2023-01-1408:59ikitommiyou can attach decode functions into any schema at the moment, e.g. have any schema control how it’s being decoded. If you want to push the control to transformer, there has to be a way to identify the schema. it can be any data pulled out of a schema.#2023-01-1409:24ikitommihere’s one way to do it using schema references (references need to be strings of qualified keywords):
(let [schema [:schema {:registry {::str-stat-schema :string}}
              ::str-stat-schema]
      decoders {::str-stat-schema (fn [x] (:value x))}
      encoders {::str-stat-schema (fn [x] {:value x})}
      compiler (fn [schema _] (when (m/-ref-schema? schema) (m/-ref schema)))
      transformer (mt/transformer
                   {:default-decoder {:compile (comp decoders compiler)}
                    :default-encoder {:compile (comp encoders compiler)}})
      decode (m/decoder schema transformer)
      encode (m/encoder schema transformer)]
  ((juxt identity decode (comp encode decode))
   {:value "kikka"}))
; => [{:value "kikka"} "kikka" {:value "kikka"}]
#2023-01-1409:27ikitommibut, it’s much easier to inject the encoders & decoders into schemas:
(def str-stat-schema
  [:string {:decode/custom (fn [x] (:value x))
            :encode/custom (fn [x] {:value x})}])

(-> [:map
     [:x str-stat-schema]
     [:y str-stat-schema]]
    (m/decode
     {:x {:value "kikka"}
      :y {:value "kukka"}}
     (mt/transformer {:name :custom})))
; => {:x "kikka", :y "kukka"}
#2023-01-1409:27ikitommihope this helps#2023-01-1617:20Dallas SurewoodThis was helpful. The plan was to make easy transformations to convert from one schema to another. The first option may be best for that, but it's very verbose. I suppose I should just make my own functions that do this manually.#2023-01-1319:49dvingoI've been working on integrating malli with react-hook-form for a bit (mostly learning react-hook-form :P ), and now have something working! https://github.com/dvingo/malli-react-hook-form You can play with it here: https://dvingo.github.io/malli-react-hook-form/ I'm sure there may be some tricky things for more complex data shapes, but this is a promising start!#2023-01-1323:04escherizeLooks familiar! I wrote a POC for something similar a long time ago. http://escherize.com/works/data-desk#2023-01-1323:06escherizeOh, I thought you were generating this form: https://github.com/dvingo/malli-react-hook-form/blob/mainline/src/app/malli_react_hook_form/entry.cljs#L44#2023-01-1323:06escherizeit’s still really cool though#2023-01-1401:53dvingono worries! that's definitely the direction I'm heading :D (generating forms from schemas) this part was just figuring out wiring of the pieces of reusable parts, next will be outputting these from the schema#2023-01-1409:30ikitommigood to see work on malli->forms! I really would like have that, have few non-complete prototypes and most likely no time to finish.#2023-01-1411:31wcalderipeI'll share a fresh thread from #clojurescript here in an attempt to connect people with similar issues https://clojurians.slack.com/archives/C03S1L9DN/p1673488701932789#2023-01-1415:07dvingojust updated to generate the form inputs from the schema https://github.com/dvingo/malli-react-hook-form/blob/8f2076cc27d23c96795c45f825c474e544f5d815/src/app/malli_react_hook_form/entry.cljs#L71#2023-01-1723:53escherizeI uploaded the lastest version of my version lately too: https://escherize.com/works/data-desk/ I think to do it right requires using a similar strategy to malli/experimental/describe.cljc where there’s a multimethod over the schema types. Might be cool to do using htmx, too.#2023-01-1801:45dvingonice! do you have the source published or no? And yep for sure, will need to deal with all the built in schemas plus nested/refs#2023-01-1415:07dvingojust updated to generate the form inputs from the schema https://github.com/dvingo/malli-react-hook-form/blob/8f2076cc27d23c96795c45f825c474e544f5d815/src/app/malli_react_hook_form/entry.cljs#L71#2023-01-1517:48pithyless> Malli requires Clojure 1.10+ and is tested against 1.10 and 1.11. There was an error reported on #C8V0BQ0M6 about a broken build, because of :as-alias being used in malli.generator (which is 1.11 only). Is the README out of date?#2023-01-1518:04ikitomminot intentional, will fix#2023-01-1606:08ikitommishould be fixed now: https://github.com/metosin/malli/pull/816#2023-01-1606:09ikitommithanks for pointing out, had missed that!#2023-01-1606:43pithylessThanks for insta-fixing it! gratitude#2023-01-1608:53moeI'm working on my first schema — what's the most concise way to specify that a value is a literal keyword (i.e. not any keyword, but a specific one)#2023-01-1608:57robert-stuttaford[:= :malli/is-awesome]#2023-01-1608:57moeHey Rob! Thanks!#2023-01-1608:57robert-stuttafordomg Moe! freakin long time dude!#2023-01-1608:58moefor real#2023-01-1608:58moemysterious ways#2023-01-1608:58robert-stuttafordglad you found The Way, as i did#2023-01-1608:58robert-stuttafordwe got up to no good back in the old days 😂#2023-01-1609:00moeI was hacking on Scheme even back then but there wasn't really anything plausible I could talk about using for work#2023-01-1609:01robert-stuttafordyeah it's come a loooong way#2023-01-1609:03moewe're still pariahs, outnumbered by COBOL developers 😕#2023-01-1610:25robert-stuttafordhaha#2023-01-1719:40escherizeWouldn’t it be nice to have a kondo hook for mx/defn? I think using https://github.com/metosin/malli/blob/master/resources/clj-kondo/clj-kondo.exports/metosin/malli/config.edn#L1 is causing problems. I was looking into the issue mentioned in https://github.com/metabase/metabase/pull/27647/files#diff-b30ee307d9c92516f1f5b912532eb62cb13c00cc267ed51df7953e9c97fb6ae8R11-R13, and started https://clojurians.slack.com/archives/CHY97NXE2/p1673983245295079#clj-kondo.#2023-01-1820:41escherizerelated: https://clojurians.slack.com/archives/CHY97NXE2/p1641752397035400#2023-01-1800:47Franco GasperinoGood afternoon. I ran into an unexpected behavior with default values, and was hoping someone could point me in the correct direction.#2023-01-1800:48Franco Gasperino
(require '[malli.core])
(require '[malli.util])
(require '[malli.transform])

(def italian-kids #{"Paul"})
(def name-opts {:min 1 :max 16})

(def decode #(malli.core/decode %1 %2 {} malli.transform/default-value-transformer))

(def children-schema
  [:map
   [:children
    [:set {:min 1 :default italian-kids}
     [:string name-opts]]]])

(decode children-schema {})
;; => {:children #{"Paul"}}

(def family-schema
  [:map
   [:dad [:string (assoc name-opts :default "Paul")]]
   [:mom [:string (assoc name-opts :default "Maria")]]
   [:children children-schema]])

(decode family-schema {})
;; => {:dad "Paul", :mom "Maria"}
(decode family-schema {:children {}})
;; => {:children {:children #{"Paul"}}, :dad "Paul", :mom "Maria"}
#2023-01-1800:49Franco Gasperinohow can i configure the second call to decode with an empty map to behave as the third call#2023-01-1815:48Franco GasperinoThe issue seems to be the nesting of :map schema definitions (in this case) when applying the default-value-transformer.#2023-01-1816:55escherizeCan’t check now, but I think you’d need a default value on the :map in children-schema#2023-01-1816:56escherize
(def children-schema
  [:map <-- add default here ?
   [:children
    [:set {:min 1 :default italian-kids}
     [:string name-opts]]]])
#2023-01-1818:26Franco GasperinoIll experiment a bit. Another option may be to leverage :default-fn. Curious if there is something more straitforward that I overlooked.#2023-01-2017:28Franco Gasperinoanswering my own question - setting a :default {} onto the :children key in the family-schema will invoke the desired behavior.#2023-01-2115:31ikitommi~That sounds like a bug, please write an issue~#2023-01-2115:33ikitommiread the example wrong, not a bug.#2023-01-1818:17Brett RowberryIf I have a fn that returns a promise with an int inside. How might I best express a promise<int> in Malli?#2023-01-1819:34valtteriAt least it can be done with :fn
(def schema [:fn (fn [x] (and (realized? x) (int? (deref x))))])

(def my-promise (promise))

(m/validate schema my-promise)
;; => false

(deliver my-promise 1)

(m/validate schema my-promise)
;; => true
#2023-01-1819:34valtteriI don’t know if there’s a better way :thinking_face:#2023-01-1819:39valtteriNOTE: this only makes sense if you know that the promise should be realized already. Another option is to omit the realized? check which will make the schema block until the promise is delivered. This may lead to weird things though.#2023-01-1819:41Brett RowberryThis looks better than what I was thinking.#2023-01-1819:43Brett RowberryThe payload isn’t actually an int, I just thought that would be easier to talk about.#2023-01-1819:44valtteriYeah, promise is just a box that can hold anything. It doesn’t really care what goes in it#2023-01-1819:44Brett RowberryIt might be nice to have a wrapper schema for promise like vector and map have.#2023-01-1819:46valtteriI think schemas are most useful on values… and promises are kinda boxes that maybe will some day contain a value. To me it’s kinda hard to think schemas on things where the outcome changes over time :thinking_face:#2023-01-1819:47valtteriMy 🧠 is limited though! Interested to hear more about your use-case if you would like to share? 🙂#2023-01-1819:49Brett RowberryI have an HTTP API that uses Reitit and Malli. I’d like to make a Clojure client (not all clients are Clojure) for it that takes advantage of the same schemas.#2023-01-1819:50Brett RowberryA synchronous client function is a solved problem. The async one is where it gets weird.#2023-01-1819:51valtteriCould malli validation to be deferred to the point in time when the value is available?#2023-01-1819:51Brett RowberryIt definitely could because I’m not even doing manual validation or instrumentation in the client, it’s just for documentation.#2023-01-1819:52valtteriOr do you want to use malli to tell “this is a function that returns a promise and the time the promise is resolved it will contain an int”#2023-01-1819:52valtteriAh, I see#2023-01-1819:52Brett RowberryI trust the service to do what the schemas say since I wrote it, and the service and client will be in the same repo#2023-01-1819:53Brett Rowberry> Or do you want to use malli to tell “this is a function that returns a promise and the time the promise is resolved it will contain an int” Yeah, that would be cool, but maybe not necessary.#2023-01-1819:57valtteriNeed to think this a little more. Currently I think that personally I’d be happy if the documentation said something like this > Returns a promise that, once realized, will contain a [:map [:x int?]] #2023-01-1820:47Brett RowberryI guess promise is more like maybe as a container - holds a single thing #2023-01-1820:48Brett RowberryExcept if the thing throws!#2023-01-1821:03escherizeThis is probably naive, but I think i’d do it like so:
(def schema [:fn (fn [x] (and (realized? x) (int? (deref x))))])

(def my-promise (promise))

(defn promise-getter [p schema]
  (let [pv @p]
    (if (mc/validate schema pv)
      pv
      (throw (ex-info "promise schema mismatch" (mc/explain schema pv))))))

(mc/validate schema my-promise)
;; => false

(deliver my-promise 1)

(try (promise-getter my-promise :string)
     (catch Exception e (ex-data e)))
;;=> {:schema :string, :value 1, :errors ({:path [], :in [], :schema :string, :value 1})}

(promise-getter my-promise :int)
;; => 1
#2023-01-2115:22ikitommiIf clj-kondo could track over promises, hinting a return value of type promise of [:map [:x int?]] could make sense as you would get static checking on how the value is accessed, e.g. needs to be unboxed first. Did not check if clj-kondo supports IDerefs.#2023-01-2115:24ikitomminot suggesting it’s a good idea, but I would:
(import '(clojure.lang IPending IDeref))

(def Promise
  (m/-simple-schema
   (fn [_ [s]]
     (let [valid? (m/validator s)]
       {:type :promise
        :min 1
        :max 1
        :pred (fn [x] (and (instance? IPending x)
                           (instance? IDeref x)
                           (if (realized? x)
                             (valid? (deref x))
                             true)))}))))

(def PromiseOfAMap [Promise [:map [:x :int]]])

(defn delivered [x] (deliver (promise) x))

(m/validate PromiseOfAMap {:x 1})
; => false

(m/validate PromiseOfAMap (delivered {:x 1}))
; => true

(m/validate PromiseOfAMap (delivered "invalid"))
; => false

(m/form PromiseOfAMap)
; => [:promise [:map [:x :int]]]
#2023-01-1821:46steveb8nQuestion: is it possible to use a custom registry with a function schema? I can’t find an example of that#2023-01-1822:58steveb8nok, I figured out a way to do it…#2023-01-1822:58steveb8n
(m/=> flow-viz [:=> [:cat (m/schema ::flow {:registry (schemas)})]
                :string])
(defn flow-viz
  [flow]
  "hi")
#2023-01-1822:58steveb8ncurious if there’s a data only way to do this e.g. with :schema#2023-01-1822:58steveb8nI couldn’t make that work#2023-01-1916:03ikitommiI think the schemas are eagerly evaluated with m/=>. I would use a custom global registry.#2023-01-1920:24steveb8nthanks. I’ll look at a custom global registry, didn’t know about that option#2023-01-1921:35steveb8n@U055NJ5CC fyi there’s a typo in the dev instrumentation docs….#2023-01-1921:35steveb8n
(plus 6)
; => 7
#2023-01-1921:35steveb8nshould be plus1#2023-01-2011:07eskosFinally took the time to release this one properly. I’d say I’m about 17% ashamed of it, which to me means it’s like the perfect time to throw it out there for people to scrutinize 🙂 From Malli’s point of view Muotti is a somewhat fancy value transformer. It’s originally made to fill a need I had and works well enough for me that I want to believe it’s useful for others as well 🤞#2023-01-2015:06ikitommihttps://github.com/metosin/malli/commit/92c64b4030102404b0e822cc43485f1de182e201 link to malli README 👍#2023-01-2016:21escherizeThis is slick — hope I find a good reason to pull it in soon#2023-01-2016:30ikitommi@U8SFC8HLP related to "can provide default values for nils", does it have benefits over using mt/default-value-transformer?#2023-01-2016:59eskosProbably not practical, since it does more than mt/default-value-transformer internally.#2023-01-2017:00eskosLets consider it another flavor, maybe one doesn't want to compose transformers, or maybe defaulting to different value in different steps makes sense...probably not, but it's ambiguous and was easy to implement originally so it's there 🙂#2023-01-2309:54mkvlrhey 👋 thanks a lot for malli, it’s very useful. We’re looking at unsing function inline schemas in our codebase and wondering about a few things: 1. what should I expect from the clj-kondo integration? Should I expect and error given the following spec & call
(mx/defn transact! [{:as system :keys [datomic]} :- [:map [:datomic some?]] tx-data :- sequential?]
  ,,,)
(transact! {} [])
2. is there a way to assert only the presense of certain keys in a map, without the some? above? Can I maybe even https://cljdoc.org/d/metosin/malli/0.10.1/doc/readme?q=keys#destructuring? 3. when do I need to call mi/instrument!? It seems I need to call it again after I change a defn with inline function schemas, is that right?
#2023-01-2309:55borkdude@U5H74UNSF 1. Try running ()
#2023-01-2309:58mkvlr@U04V15CAJ that’s for the clj-kondo integration?#2023-01-2309:59borkdudealso for automatic instrumentation#2023-01-2309:59mkvlrsee the docstring now but not seeing an effect yet#2023-01-2310:00borkdudehttps://github.com/metosin/malli/blob/cfbc8eaa05bedb1e7e7b132d52bbf2e61b143f92/src/malli/dev.clj#L15#2023-01-2310:00borkdudeyou need to make a few edits before it has an effect, to re-trigger clojure-lsp / clj-kondo#2023-01-2310:00mkvlror I see an output on the REPL buffer now#2023-01-2310:03borkdudeI'll try locally#2023-01-2310:04mkvlrbut calling the function with invalid args doesn’t error, only after I call (mi/instrument!) again#2023-01-2310:04borkdudeworks here:#2023-01-2310:05mkvlrvery cool#2023-01-2310:06borkdudeI only have to re-evaluate the defn and then make 1 additional edit to re-trigger linting#2023-01-2314:22ikitommire-evaluating the mx/defn should automatically re-create clj-kondo configs when dev/start! is running.#2023-01-2316:13escherize@U5H74UNSF — Do you mean that calling a function with mx/defn with the wrong args doesn’t immediately fail?#2023-01-2316:21escherizeI had that problem, and fixed it by having the instrumentation call baked into the macro: https://github.com/metabase/metabase/blob/master/src/metabase/util/malli.clj#L53-L78#2023-01-2316:22escherizethen it behaves like s/defn. I think there was talk of including somethng like this (mx/defn ^:always-check …) but I havn’t kept up with it.#2023-01-2316:25escherizeIF we are talking about adding options to mx/defn, tho. My wish list would have: 1. “always check args and return value.” 2. It’d be cool to have a way to say “just make the shell of a function, which takes the args, checks them against the schema, and returns a generated value from the schema.” I know it’s mechanically not difficult, just needs to be hooked up. Maybe I should make an issue.#2023-01-2316:42ikitommiIssue welcome, agree that both would be good additions.#2023-01-2317:32escherizeI broke it into 2, since I’ve written #1 already. Had some open questions though. 1) https://github.com/metosin/malli/issues/823 2) https://github.com/metosin/malli/issues/824#2023-01-2317:58dvingoPut together a PR for 1) here https://github.com/metosin/malli/pull/702/files#diff-08fdf5164d5d2562ba6f3e187911a9db8086c3d06a3d048cd8045818b2d960e5R94 main motivation at that point was for better cljs DX, which isn't needed anymore, but having a form that is always instrumented is definitely useful#2023-01-2314:07geraldodevIs it possible to compare schemas ?#2023-01-2314:17ikitommidid you check malli.util/equals?#2023-01-2314:17ikitommiundocumented, not super performant, just checks the forms are equals…#2023-01-2314:19geraldodevNo, I've looked into documentation trying to find something like equals. Thank you.#2023-01-2314:31geraldodevShouldn't (mu/equals [:map [:foo [:maybe :string]] [:bar :some]] [:map [:bar :some] [:foo [:maybe :string]]]) be considered equal ?#2023-01-2314:40ikitommiif ordering matters, they are not equal. But I see the value in having a utility for deep-equals, ignoring order.#2023-01-2314:40ikitommisee https://github.com/metosin/malli/issues/82#2023-01-2314:41ikitommibut, no such too at the moment.#2023-01-2314:43geraldodevI was manually running mp/provide to infer the type to inject on reitit to get on the other side with some swagger generator to typescript, so a not positional equals would help.#2023-01-2316:25escherizeIF we are talking about adding options to mx/defn, tho. My wish list would have: 1. “always check args and return value.” 2. It’d be cool to have a way to say “just make the shell of a function, which takes the args, checks them against the schema, and returns a generated value from the schema.” I know it’s mechanically not difficult, just needs to be hooked up. Maybe I should make an issue.#2023-01-2414:21Noah BogartI've introduced instrumentation at my workplace, and someone brought up the issue of the messiness of .clj-kondo/metosin/malli-types/config.edn updating between different branches/PRs and whether we should commit the file or not. We're calling (mdev/start!) in both dev and testing environments, so I believe that's generating the files fresh every time. Could we get away with .gitignoring the file and having it be generated by all users and the CI pipeline?#2023-01-2416:22escherizeWe https://github.com/metabase/metabase/blob/master/.gitignore#L87-L88 it#2023-01-2415:19Noah BogartFor those using function instrumentation in prod, do you run (mdev/start!) somewhere in your start-up sequence?#2023-01-2415:56ikitommimalli.dev ns for development use, for prod, see malli.instrument (used by malli.dev).#2023-01-2415:56Noah BogartAh I see, okay cool. Thanks#2023-01-2510:15gravWhat's the status of converting json-schemas to malli?#2023-01-2510:15gravI saw @UK0810AQ2 mentioning something about a proof-of-concept here: https://clojurians.slack.com/archives/C053AK3F9/p1673410476605509?thread_ts=1673402053.813639&amp;cid=C053AK3F9#2023-01-2510:17Ben Slesshttps://gist.github.com/bsless/fc0c48c2f983cf0243f0bd72c44d1d0bb I need to flesh this out#2023-01-2510:19delaguardo404, looks like it is private gist#2023-01-2511:00grav@UK0810AQ2 Would you mind sharing what you've got?#2023-01-2514:54Ben SlessI'll fix it when I get home#2023-01-2517:50Ben Sless@U04V4KLKC @U052XLL3A try now https://gist.github.com/bsless/fc0c48c2f983cf0243f0bd72c44d1d0b#2023-02-0119:17Ben Sless@U02CV2P4J6S here#2023-01-2512:17flowthingGiven [:map {:foo {:default :bar}}], does Malli have a API for getting the default value?#2023-01-2512:27ikitommithat’s an empty map with one property :foo :thinking_face:#2023-01-2512:28flowthingGrah, sorry, typo. 🙂#2023-01-2512:28flowthing[:map [:foo {:default :bar} keyword?]]#2023-01-2512:34juhoteperihttps://github.com/metosin/malli#value-transformation mt/default-value-transformer#2023-01-2512:36flowthingHmm, right, I know about that, but I was wondering whether I could introspect the schema to figure out the default value for a given key. But now that you mention it, using value transformation is probably the better route. 👍 Thanks!#2023-01-2512:40juhoteperi(:default (m/properties (mu/get map-schema :foo))) might work#2023-01-2512:41flowthingI tried that earlier, but it doesn’t seem to:
(malli.core/properties (malli.util/get [:map [:foo {:default :bar} keyword?]] :foo))
;;=> nil
#2023-01-2512:42juhoteperiDid you check what just mu/get returns?#2023-01-2512:42flowthingYeah, it returns keyword?, so it’s not suprise it doesn’t work. I also tried find:
(malli.util/find [:map [:foo {:default :bar} keyword?]] :foo)
;;=> [:foo {:default :bar} keyword?]

(malli.core/properties (malli.util/find [:map [:foo {:default :bar} keyword?]] :foo))
;;=> Execution error (ExceptionInfo) at malli.core/-exception (core.cljc:138).
;;   :malli.core/invalid-schema
but that throws.
#2023-01-2512:43juhoteperiOh right, and the default property is in the map schema key props#2023-01-2512:45flowthingI mean I can get to it using find, but it doesn’t feel quite right. 🙂#2023-01-2512:54ikitommiideas most welcome for a better api, currently mu/find or m/entries on the map to iterate all entries.#2023-01-2512:56flowthingGood to know, thanks. 👍 I think coercing the map to contain the default values is the right way to go in my case, though.#2023-01-2603:54escherizeshould mx/defn do type hinting automatically?#2023-01-2604:31escherizemx/defn doesn’t grab metadata off the var, just from the metadata position, unless Im missing something. It’d be nice to pass metadata into it from the name, IMO.#2023-01-2606:52ikitommi> mx/defn doesn’t grab metadata off the var, just from the metadata position, unless Im missing something. It’d be nice to pass metadata into it from the name, IMO. I noticed that too. But, that’s how clojure works right? but I’m onboard of making mx/defn better. auto-type-hinting would require mapping from schemas to types, right?#2023-01-2606:54escherizeI have a bunch of mx/defn improvements.#2023-01-2606:55escherizeOne I just thought of is, using catn instead of cat so [:=> [:cat :int :int] :int] could become [:=> [:catn [:a int] [:c int]] :int]#2023-01-2606:55escherizethe ones I have right now, are :no-throw, :gen, :seed, and :size#2023-01-2606:56escherizeI am calling mg/generate — but I think there’s a way to use -strument to do the generation#2023-01-2606:56escherizeidk which is better#2023-01-2606:57escherizefrom the function schema doc:
(def pow-gen
  (m/-instrument
    {:schema [:function
              [:=> [:cat :int] [:int {:max 6}]]
              [:=> [:cat :int :int] [:int {:max 6}]]]
     :gen mg/generate}))
#2023-01-2606:57ikitommiI guess with symbols, so:`[:=> [:catn ['a int] ['c int]] :int]`#2023-01-2606:57escherizesymbols!! 😍#2023-01-2606:58ikitommiyeah, the name can be anything#2023-01-2606:52escherizeYeah. Does that mapping exist already? It’s mostly trivial. ~One question is should we it let one add a custom typehint?~#2023-01-2606:54escherizeno need to allow that unless people want it.#2023-01-2606:59ikitommino such mappings yet. I have the schema-type -> type mapping in top of the backlog, e.g. int? is a :int. simplifies things like JSON Schema mappings would do it with (java/js) type hints too.#2023-01-2607:04escherizeThat is something that will be really good to have#2023-01-2610:30Karel MiarkaHello, please how to add a custom generator? Cannot figure out: [:fn {:generator-fn #?(:clj #(Date.) :cljs #(js/Date.))} (fn [x] (instance? #?(:clj Date :cljs js/Date) x))]#2023-01-2611:16ikitommisearch for :gen/gen from the README (requires a generator, not sure if plain function is valid here.#2023-01-2611:16ikitommishould there be :gen/fn :thinking_face:#2023-01-2611:27dharrigan@U04M0JEU88Z here's a little example of using a custom generator: #2023-01-2612:46Karel MiarkaThank you both for you replies. I have fortunately found clojure.core.inst? fn to be a better replacement for my case. Still not sure how to fix my previous trouble, but the answer can be probably found by studing clojure.test.check.generators and than using :gen/gen .#2023-01-2614:24Noah Bogartstyle/usage/architecture question: we're using Malli as internal schemas (instrumentation/documentation) and then also relying on the json schema transformation for input/output validation (our system has an older custom json schema builder, lol). Combined, this means our schemas are a little unwieldy and have a lot of repetition. For example, for a single POST foo endpoint, we have • NewFoo schema • post-foo-payload which is the json schema transformation of the NewFoo schema • private post-foo-decoder which is standalone to get speed from m/decoderdecode-foo-params that uses post-foo-decoder to decode the payload • PostFooResponse schema (which is thankfully just (mu/assoc NewFood :extra-key :uuid)) • private post-foo-response-encoder which is standalone to get speed from m/encoderencode-post-foo-response that uses post-foo-response-encoder to do the final-step encoding/extra key stripping before calling (compojure/response resp) • and finally post-foo-response which is the json schema transformation of the PostFooResponse schema That's a lot and some of it comes from using Malli to generate json schema schemas, I know, but there's still a fair amount of boilerplate happening. Does anyone have ideas for how to lessen the code footprint for situations like this?#2023-01-2615:05ikitommiif that is all for a single POST, It’s a lot. There are two good ways to organize schemas: 1. Vars a. like plumatic, (def NewFoo (m/schema …)) b. simple, usually non-qualified keys 2. Global Registry a. like spec, description here: https://github.com/metosin/malli#mutable-registry b. mutable = evil, qualified keys. for serving requests over web, the boilerplate of dealing with optimized encoders & decoders can be pushed into frameworks, e.g. reitit + malli here: https://github.com/metosin/reitit/blob/master/examples/ring-malli-swagger/src/example/server.clj#L57-L81. btw, nowadays malli caches automatically validators, explainers and generators but NOT transformers. so this is almost as fast as manually creating a validator:
(def Foo (m/schema [:map [:x :int]]))

;; caches the validator on first invocation into schema instance
(m/validate Foo {:x 1})
#2023-01-2615:14Noah Bogart> nowadays malli caches automatically validators, explainers and generators Really? That's great news. > can be pushed into frameworks that's what our json schema is meant to handle, but switching over entirely to malli is long and hard and there's no desire to move off of compojure, so right now i'm stuck carving out a small corner of sanity in the much larger code base.#2023-01-2615:31pithyless> • private post-foo-decoder which is standalone to get speed from m/decoder > • decode-foo-params that uses post-foo-decoder to decode the payload I've usually created closed over functions that internally create the decoder.
(def decode-foo-params [..] (let [decoder (..)] (fn [..] (decoder ...))
#2023-01-2615:31pithylessI've also created orchestration functions that deal with the repetitions of an operation (and you can more declaratively just pass in a map of {:my-schema ,, :post-decoder-fn ,,, :etc})#2023-01-2615:37pithylessArne also had a talk about data-driven malli which I think can serve as inspiration on how to generate more of this stuff based on some initial schemas. https://www.youtube.com/watch?v=ww9yR_rbgQs#2023-01-2615:53Noah Bogartvery cool, thanks for the talk. watching it now#2023-01-2616:35Ben SlessIf you care about performance then you should consider dropping compojure, otherwise you can borrow the coercion middlewares from reitit#2023-01-2616:49Noah Bogarthah I would love to drop compojure. getting organizational buy-in is quite challenging, sadly#2023-01-2617:04Karel MiarkaHello, is there a way and example how to define Malli schema for functions with optional named arguments like this:
(defn my-named-args-fn [& {:keys [x y z]}]
  (println x y z))

(my-named-args-fn :x 1 :y 2)
#2023-01-2617:34ikitommino simple built-in for that, but:
(defn my-named-args-fn [& {:keys [x y z]}]
  (println x y z))

(malli.destructure/infer #'my-named-args-fn)
;[:=>
; [:cat
;  [:altn
;   [:map [:map
;          [:x {:optional true} :any]
;          [:y {:optional true} :any]
;          [:z {:optional true} :any]]]
;   [:args [:* [:alt 
;               [:cat [:= :x] :any]
;               [:cat [:= :y] :any]
;               [:cat [:= :z] :any] [:cat :any :any]]]]]]
; :any]
#2023-01-2618:09Noah Bogartshould that be an :or? maybe I don't remember how that destructuring works#2023-01-2623:39escherizeto add, if you use mx/defn on that shape, the destructuring schema will be inferred. ( I think ).#2023-01-2623:40escherizeyea.#2023-01-2623:40escherize
(require '[malli.experimental :as mx])

(mx/defn my-named-args-fn [& {:keys [x y z]}]
  (println x y z))

(:schema (meta #'my-named-args-fn))

;; => [:=>
;; =>  [:cat
;; =>   [:altn
;; =>    [:map
;; =>     [:map
;; =>      [:x {:optional true} :any]
;; =>      [:y {:optional true} :any]
;; =>      [:z {:optional true} :any]]]
;; =>    [:args
;; =>     [:* [:alt
;; =>          [:cat [:= :x] :any]
;; =>          [:cat [:= :y] :any]
;; =>          [:cat [:= :z] :any] [:cat :any :any]]]]]]
;; =>  :any]
#2023-01-2623:41escherizeyou could have a function that return that shape ^#2023-01-2706:23ikitommi@UEENNMX0T :alt & :altn are the or for sequences. That should work ok.#2023-01-2706:26ikitommibtw, like with mostly everything else in malli, you can configure the dest inferring with options, “closed map with required keys”:
(md/infer #'my-named-args-fn {::md/closed-maps true, ::md/required-keys true})
;[:=>
; [:cat
;  [:altn
;   [:map [:map {:closed true}
;          [:x :any]
;          [:y :any]
;          [:z :any]]]
;   [:args [:* [:alt
;               [:cat [:= :x] :any]
;               [:cat [:= :y] :any]
;               [:cat [:= :z] :any]]]]]]
; :any]
#2023-01-2706:27ikitommiit would be easy to build a custom schema to cover the both cases (map or varargs), e.g.:
[:=>
 [:cat
  [:kvmap
   [:x :any]
   [:y :any]
   [:z :any]]]
 :any]
… that would work the same. haven’t needed so it’s not there yet.
#2023-01-2708:06Karel MiarkaThanks a lot. I will play with that and would like to create a fn to simplify the declaration.#2023-01-2620:59Michael GardnerI have some namespaces that use :malli/schema function metadata, with (m.inst/collect!) as the final line in each namespace. That stopped working in 0.9.0:
Syntax error (ClassNotFoundException) compiling at (me/foo.clj:180:1).       
me.foo
Was this ever intended to work?
#2023-01-2718:35Michael Gardner@U055NJ5CC I don't see anything about this in the 0.9 changelog. Should I whip up a full repro? I'd thought there were others doing this, but maybe I'm the only one?#2023-01-2811:32ikitommithere should not be any breakage. Please create an issue with minimal repro.#2023-01-2811:32ikitommiworks here ok :thinking_face:
(ns demo)

(defn plus
  "adds two numbers together"
  {:malli/schema [:=> [:cat :int :int] :int]}
  [z y]
  (+ z y))

(require '[malli.instrument])
(malli.instrument/collect! {:ns 'demo})
;#{{:clj {demo {plus {:schema [:=> [:cat :int :int] :int]
;                     :ns demo, :name plus}}}}}
#2023-02-0109:49Bart KleijngeldFor what it's worth: I'm having the same problem. Did you find out what was the matter @U01R1SXCAUX?#2023-02-0110:02Bart KleijngeldTo be clear: • (mi/collect!) does not work • (mi/collect! {:ns 'demo}) does Upon closer inspection, the 0-arity call defaults to using *ns* as namespace. In my case its value was different from what I expected during my REPL session. Maybe the same for you Michael.#2023-02-0111:20ikitommiUgh, that doesn't look right. Not using the 0-arity, it's clearly broken.#2023-02-0111:20ikitommiIssue welcome, should be easy to fix#2023-02-0112:00Bart Kleijngeldhttps://github.com/metosin/malli/issues/834#2023-02-0112:43ikitommifixed in master.#2023-02-0117:53Michael Gardnerthank you!#2023-01-2621:02Varghiz cljHi, I'm looking for component/solution where I can transform malli schema to clerk to document the schema/spec in html format... This is just a start but ultimate need is to convert it to something I can publish in confluent wiki#2023-01-2706:35ikitommirendering schemas + also schema editors would be awesome.#2023-02-1114:35Varghiz cljThank you @U055NJ5CC#2023-01-2623:25escherizedumb question: why not use this as a malli-schema malli schema? [:fn m/schema] I guess you can’t use it to generate, but it’ll work for certain things.#2023-01-2706:31ikitommiyes, that works if you rely on global options.#2023-01-2700:02escherizeI’d like to make https://malli.io/?value=(%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D)&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D fail (since the outermost sequence is a list not a vector) It is valid, but I want to force the sequences enforced by :* to be vectors. what’s a good way to do that?#2023-01-2706:34ikitommicurrently https://malli.io/?value=(%3Adiv%20%7B%3Aclass%20%5B%3Afoo%20%3Abar%5D%7D%20%5B%3Ap%20%22Hello%2C%20world%20of%20data%22%5D)&amp;schema=%5B%3Aschema%0A%20%7B%3Aregistry%20%7B%22hiccup%22%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anode%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aand%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Acatn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aname%20keyword%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprops%20%5B%3A%3F%20%5B%3Amap-of%20keyword%3F%20any%3F%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Achildren%20%5B%3A*%20%5B%3Aschema%20%5B%3Aref%20%22hiccup%22%5D%5D%5D%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20vector%3F%5D%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aprimitive%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aorn%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anil%20nil%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Aboolean%20boolean%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Anumber%20number%3F%5D%0A%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%20%5B%3Atext%20string%3F%5D%5D%5D%5D%7D%7D%0A%20%22hiccup%22%5D. Allowing type hints to everything would allow [:catn {:type :vector} …] which would be better.#2023-01-2718:34escherizeI misread your answer, thanks :}#2023-01-2812:51pithylessCan I define a generator for [:map [:a :string] [:b string]] such that I can :gen/fmap the :b based on the generated value of :a ?#2023-01-2812:53pithylesseg. if using faker I can generate:
{:company-name "Acme Inc"}
but I'd like to slugify the name via some function:
{:company-name "Acme Inc.", :slug "acme"}
#2023-01-2812:56ikitommiso, :gen/fmap is not good enough here?#2023-01-2812:56pithylessShould I :gen/fmap on the :map itself?#2023-01-2812:57ikitommiyes, I would do that#2023-01-2812:57pithylessThat makes sense. Thanks for the sanity check :)#2023-01-2817:05nonrecursivehey y’all, is there a way to create a map spec where all the keys are optional, but values must have one of the keys to be valid?#2023-01-2817:06nonrecursiveeg given [:map [:a {:optional true}] [:b {:optional true}]], then {:a :foo} would be valid but {:c :foo} would not#2023-01-2817:07nonrecursivei can think of [:or [:map [:a]] [:map [:b]]] but that seems clunky?#2023-01-2818:04pithylesswell, it sounds like you don’t want optional but you’re actually modeling a union type - I’d go with :multi or :or - but if that feels clunky you probably need an :and where you first spec the map and then check with custom fn that at least one of the expected keys exists.#2023-01-2818:07pithyless(You’ve still got things like :merge to help you pull out the repetitive parts of the individual unions)#2023-01-2913:53nonrecursivethank you!#2023-01-2903:58dvingofigured out a way to get clj-kondo config output persisted during dev for cljs function schemas! https://github.com/metosin/malli/pull/829#2023-01-3018:09jasonjcknhas anyone written a generic malli schema to spec converter function i can steal? as i transition a code base over to malli, would be a helpful tool #2023-01-3018:12pithylessI remember this thread: https://clojurians.slack.com/archives/CLDK6MFMK/p1663090650431439 No idea if anything changed from that time.#2023-01-3018:15jasonjcknThanks 🙏 this looks interesting from the thread and exactly my use case https://github.com/Blasterai/malli-datomic#2023-01-3018:16emilaasaI've got this dream as well 🙂#2023-02-0323:04kennySpec->Malli 😉 https://github.com/kennyjwilli/specalli#2023-01-3023:32escherizeI’m trying to do generative testing on some functions created with mx/defn. Is there a good way to see how many things match a filter for mi/check?
(mi/check {:filters [(mi/-filter-var #{#'my/wacky-fxn})]})
I’d like to double check that I actually got a var. Ideally check would return something like {my.ns/wacky-fxn :ok}. Does this exist somewhere already? It looks like -strument calls into :mode after the var lookup doesn’t happen. I can build a similar thing myself in that case
#2023-01-3023:46escherizeI figured it out, I can pass my own mode function into -strument!#2023-01-3104:09escherizeI ended up just passing the var into this
(defn- check-fn! [fn-var & [iterations]]
  (let [iterations (or iterations 5000)]
    (if-let [result ((mg/function-checker (:schema (meta fn-var)) {::mg/=>iterations iterations}) @fn-var)]
      result
      {:pass? true :iterations iterations})))
which will throw if the var isn’t resolvable
#2023-01-3113:11flowthingWhat’s the correct way to handle this?
(malli/explain [:map [:tag [:enum :foo]]] (clojure.data.xml/sexp-as-element [:foo]))
;;=> Execution error (AbstractMethodError) at clojure.data.xml.node.Element/entryAt (node.cljc:-1).
;;   Method clojure/data/xml/node/Element.entryAt(Ljava/lang/Object;)Lclojure/lang/IMapEntry; is abstract
Do I need to walk the return value of sexp-as-element, transforming clojure.data.xml.node.Elements into maps before handing them to Malli, or is there a smarter way?
#2023-01-3113:22flowthingSomething like this, perhaps?
(malli/explain Schema (malli/decode Schema xml (malli.transform/transformer {:name :my-custom-thing})))
#2023-01-3114:13ikitommi:thinking_face: looks like the Element is not implementing clojure.lang.Associative, which malli programs against. If you could PR that into the data.xml, I should just work.#2023-01-3114:15ikitommiotherwise, you need to transform it, works, but much slower, e.g.
(m/coerce 
  [:map {:type 'clojure.data.xml.node.Element} [:tag [:enum :foo]]]
  {:tag :foo}
  custom-transformer-to-transform-the-thing)
#2023-01-3114:16ikitommior:
(m/coerce 
  [:map {:decode/xml ...} [:tag [:enum :foo]]]
  {:tag :foo}
  (mt/transformer {:name :xml}))
#2023-01-3117:03flowthingThanks! Yeah, that’s pretty much what I ended up with. Performance isn’t critical in this case, so this’ll do just fine. 👍#2023-02-0208:45flowthingThis solution doesn’t work with nested [:map ,,,]s, unfortunately:
(def Schema
    [:map {:decode/xml (partial into {})}
     [:tag [:enum :foo]]
     [:content
      [:+ [:map {:decode/xml (partial into {})}
           [:tag [:enum :bar]]
           [:content [:+ string?]]]]]])
  ;;=> #'user/Schema

  (malli/coerce Schema
    {:tag :foo :attrs {} :content [{:tag :bar :attrs {} :content ["1"]}]}
    (transform/transformer {:name :xml}))
  ;;=> {:tag :foo, :attrs {}, :content [{:tag :bar, :attrs {}, :content ["1"]}]}

  (try
    (malli/coerce Schema
      {:tag :foo :attrs {} :content [{:tag :BAD :attrs {} :content ["1"]}]}
      (transform/transformer {:name :xml}))
    (catch clojure.lang.ExceptionInfo ex
      (-> ex ex-data :data :explain error/humanize)))
  ;;=> {:content [{:tag ["should be :bar"]}]}

  (try
    (malli/coerce Schema
      (xml/sexp-as-element [:foo [:bar "1"]])
      (transform/transformer {:name :xml}))
    (catch clojure.lang.ExceptionInfo ex
      (-> ex ex-data :data :explain error/humanize)))
  ;;=> {:tag :foo, :attrs {}, :content [{:tag :bar, :attrs {}, :content ["1"]}]}

  (try
    (malli/coerce Schema
      (xml/sexp-as-element [:foo [:BAD "1"]])
      (transform/transformer {:name :xml}))
    (catch clojure.lang.ExceptionInfo ex
      (-> ex ex-data :data :explain error/humanize)))
  ;;=> Execution error (AbstractMethodError) at clojure.data.xml.node.Element/entryAt (node.cljc:-1).
  ;;   Method clojure/data/xml/node/Element.entryAt(Ljava/lang/Object;)Lclojure/lang/IMapEntry; is abstract
Dunno whether custom-transformer-to-transform-the-thing would solve this problem, but I’m unclear on how I’d implement that, since I can’t find any documentation or examples on how to implement custom transformers like that.
#2023-02-0208:59ikitommiSomething like:
(def elements-to-maps-transformer
  "transforms all Elements into maps"
  (mt/transformer
   {:decoders {:map #(cond->> % (instance? Element %) (into {}))}}))
#2023-02-0208:59ikitommi1. apply only to :map schemas 2. run a function that checks if it’s an Element, then make it a map#2023-02-0209:03ikitommi… and you can compose, the order matters:
(mt/transformer
  elements-to-maps-transformer ;; this first 
  (mt/json-transformer)) ;; ... so these see maps, not emelents
#2023-02-0209:07flowthingThanks, I was missing the :map bit. 👍 Still no dice, though, unfortunately: Malli only applies the transformation to the root value.#2023-02-0209:08flowthingI think the simplest solution in this case is for me to walk the thing sexp-as-element returns and turn Elements to maps before handing the value to Malli.#2023-02-0209:11ikitommioh, true. into (into {} x) is not recursive. change it to (clojure.walk/postwalk #(cond->> % (instance? Element %) (into {}))) and should work#2023-02-0209:31flowthingYep, could do that, but I don’t think that has any benefit over just transforming the value beforehand in this case.#2023-02-0209:31flowthingFWIW, https://ask.clojure.org/index.php/12625/clojure-element-doesnt-implement-clojure-associative-entryat.#2023-01-3117:41shem
{:static1 "foo"
:static2 "bar"
:var-1234 "baz"
:var-9725 "euromokko"}
#2023-01-3117:43shemif we have maps like these, where the static* keys are known and the numbers ending the var* keys are not known and may be any integer, is it possible to construe a schema that also covers the var* keys?#2023-01-3117:52respatializedyou could use :map-of and a predicate schema to constrain the names of the var-* keys, then :merge with the static keys#2023-01-3118:09shemright...i'll experiment with that. thanks!#2023-01-3118:56shem
(def samppeli {:static1 "foo"
                 :static2 "bar"
                 :var-1234a "baz"
                 :var-9725 "euromokko"})
  (def varskeema (m/schema [:map-of #"^var-\d+$" :string]))
  (def staticskeema (m/schema [:map [:static1 :string]
                               [:static2 :string]]))
  (def skeema  (mu/merge varskeema staticskeema))
#2023-01-3118:58shemthis validates ok even though it shouldn't, probably because maps are open by default. i can't seem to find the right place for {:closed true}#2023-01-3119:19respatializedTry using mu/`update-properties` on the results of mu/merge#2023-01-3120:08escherizeor mu/closed-schema on it#2023-01-3117:45shemi couldn't find pattern matching in the keys#2023-01-3120:28escherizeWe are gonna need some more info if you are trying to get help.#2023-01-3121:40dvingowith the experimental syntax is there a way to specify a unique return type per arity? I only see one in the tests/examples.
(mx/defn a-fun  ;; :- [:int {:min 0}] <--- instead of this
  "docstring"
  ([x :- [:int {:min 0}]] :- :int (inc x))
  ([x :- [:int {:min 0}], y :- :int] :- [:int {:min 10}] (+ x y))
  ([x :- [:int {:min 0}], y :- :int & zs :- [:* :int]] :- :int  (apply + x y zs)))
something like that, I'm not sure where the syntax would go. With :function schema annotations this is supported, so it's just a matter of where the syntax lives and how it's transformed.
#2023-01-3121:57escherizeI don’t think so. I was looking at that the other day. notice that in the tests there is no such thing.#2023-01-3121:58escherizeyou can of course use m/=> to get that behavior. but I don’t think mx/defn can#2023-01-3122:39dvingocool cool - thanks for confirming!#2023-01-3123:40escherizeBeen noodling on this one for a while. I think I have a solution, but would be nice to hear thoughts: I want to conditionally validate fxn annotations for input/output based on the value of an atom (a switch to turn it off or on). I am thinking of 3 modes: 1. usually : validate iff @*validate-schemas. then either of these will override *validate-schemas: 2. ^:never-validate : never validates 3. ^:always-validate : always validates spoiler: Currently I am emitting a call to instrument!, but I am thinking that it makes sense to use -strument, and use a function in mode… Then again it’s probably better to just require the user to specify their auto validation wants before we start defining functions.#2023-02-0116:47escherizeJust saw the PR for this! Looking forward to seeing some more improvements on mx/defn. 🆒#2023-02-0101:26geraldodev"I'm looking at the ring-malli-lite-swagger and ring-malli-swagger of reitit and the differences in the server.clj are just in the syntax. So it looks like reitit recognizes the lite syntax without configuration. It's very nice that we can use assoc/dissoc/merge in plain Clojure and reduce clutter."#2023-02-0112:37ingesolHow would you write a function schema for (defn foo [a & {:keys [b c}] ...) ?#2023-02-0112:42delaguardoYou could use malli.destructure to infer a schema.
user=> (require '[malli.destructure :as md])
nil
user=> (defn foo [a & {:keys [b c]}])
#'user/foo
user=> (md/infer #'foo)
[:=> [:cat :any [:altn [:map [:map [:b {:optional true} :any] [:c {:optional true} :any]]] [:args [:* [:alt [:cat [:= :b] :any] [:cat [:= :c] :any] [:cat :any :any]]]]]] :any]
#2023-02-0114:04ingesolGreat thanks, didn’t remember that one#2023-02-0117:17Bart KleijngeldDoes someone know if there's a way to have :malli/schema metadata on functions be rendered into documentation by Codox?#2023-02-0117:18Bart KleijngeldSo the function schema would be visible in the docs for that function. Even a plain text dump would be nice.#2023-02-0117:27Ben SlessYou could probably do it in codox by configuring which metadata it looks at#2023-02-0117:29Bart KleijngeldYeah you're probably right. And I expect whatever theme I'll be using must also be extended to show that metadata.#2023-02-0117:36escherizeIf you need to produce strings / explainations there’s malli.experimental.describe/describe too!#2023-02-0117:36escherizehttps://github.com/metosin/malli#description#2023-02-0117:40escherizeWe are using that to generate some simple docs ^#2023-02-0119:00Bart KleijngeldThat may come in handy, thanks :)#2023-02-0117:29frankhas any progress been made on this front?#2023-02-0119:00Benjaminhttps://json-schema.org/ is there tool that generates malli from json schema and would this be useful? I was doing something with google docs api in the past and it might have been mildly useful#2023-02-0119:11Noah BogartThere's https://github.com/metosin/malli/issues/54 about it, along with this https://github.com/metosin/malli/pull/211. Maybe you or someone else could pick it up and finish it? I'd also love to have this functionality but my job only allows me enough time to fix a bug here or there.#2023-02-0119:16Ben SlessThere's also a gist I shared some time ago in malli, need to dig it up#2023-02-0213:58Ben SlessConundrum: how can we make sure a user's pred implementation in custom simple schema is correct? Is it enough that it returns truthy value, or is boolean strictly required? The validator doc string guarantees it returns a function which returns a boolean, but it's pretty easy to violate this. Thoughts?#2023-02-0214:12respatializedclearly, the only solution is a self-describing meta-schema for all Malli schemas 😉 #2023-02-0214:14Ben SlessAND malli dev instrumentation running in the background to catch everything#2023-02-0214:14Ben SlessThat's not a bad idea, btw#2023-02-0214:14Ben SlessA hassle, but could work#2023-02-0215:30dvingothanks to Thomas Heller, I was able to simplify the implementation of clj-kondo support in cljs https://github.com/metosin/malli/pull/833 (this supersedes my prior PR for this) this also updates the general instructions for use with cljs to use preloads which vastly simplifies dev vs release config#2023-02-0215:42ikitommiThis is brilliant!! I'll check it tomorrow 🙇#2023-02-0216:55pithylessVery cool, thanks for working on this! I'm not currently doing much CLJS work, but I remember there were some rough edges in getting the dev experience right.#2023-02-0219:53dvingofor sure! I've definitely learned a ton about shadow-cljs and the cljs compiler in general working on this. I think the big improvement since the early implementation is instrumenting at runtime (also a T Heller suggestion :) ) that happened with this PR https://github.com/metosin/malli/pull/755 it's more deterministic as you're not having to think about what is a macro and what is not#2023-02-0222:11escherizeNice!#2023-02-0216:09Ngoc KhuatHi guys, the error/fn key takes a function with 2 arities, what is the second argument for?#2023-02-0221:48ikitommioptions, https://github.com/metosin/malli/blob/master/src/malli/error.cljc#L136-L139#2023-02-0221:38nyor.trHi, how can I make a schema which encodes a boolean to a string and vice versa? I tried:
(def int-boolean-schema
  (m/-simple-schema
   (fn [_ _]
     {:type :int-boolean
      :type-properties {:decode/string
                        (fn [s]
                          (when-not (clojure.string/blank? s)
                            (get {"0" false "1" true} s)))
                        :encode/string
                        (fn [v]
                          (prn "val" v)
                          (if (true? v)
                            "1"
                            "0"))}})))
Decoding and encoding true works as expected, but encoding false returns nil without entering the :encode/string functions.
;; correct:
(m/decode [:int-boolean] "0" mt/string-transformer)
=> false

;; correct:
(m/decode [:int-boolean] "1" mt/string-transformer)
=> true

;; correct:
(m/encode [:int-boolean] true mt/string-transformer)
=> "1"

;; wrong, it should be "0":
(m/decode [:int-boolean] false mt/string-transformer)
=> nil
#2023-02-0222:41hanDerPederShouldn't this work?
(m/schema [:map {:registry {:my/int [:int {:doc "My int, it's very special"}]}}
           [:key [:my/int {:foo "bar"}]]])
throws invalid schema exception
:malli.core/invalid-schema
   {:type :malli.core/invalid-schema,
    :message :malli.core/invalid-schema,
    :data {:schema [:int {:doc "My int, it's very special"}]}}
If I remove the properties from either the registry or the map type it's fine. Maybe properties should be merged?
#2023-02-0300:17dvingoI think I ran into something similar before (I didn't figure out why it fails yet):
(m/schema [:comment {:some :prop}] 
  {:registry (merge (m/default-schemas) {:comment :int})})
=> [:int {:some :prop}]
does not work:
(m/schema [:comment {:some :prop}] 
  {:registry (merge (m/default-schemas) {:comment [:map [:a :int]]})})

Execution error (ExceptionInfo) at malli.core/-fail! (core.cljc:138).
:malli.core/invalid-schema {:schema [:map [:a :int]]}
#2023-02-0309:25hanDerPederI suspect stepping through malli.core/schema with these inputs will shed some light on this. I'll take a look when time permits.#2023-02-0803:09dvingoI ran into this again and looked into what's going on. have a fix here: https://github.com/metosin/malli/pull/848#2023-02-0808:33hanDerPederNice!#2023-02-0815:31dvingoso it isn't supported to use the vector form in the registry. Will need to change strategies in our apps to handle this desired behavior#2023-02-0815:32dvingoone idea is to use :schema
(m/schema [:schema {:some :prop} :comment]  
  {:registry (merge (m/default-schemas) {:comment [:map [:a :int]]})})
#2023-02-0819:16dvingoalso found that add a :ref works
(m/schema [:map {:registry {:my/int [:int {:doc "My int, it's very special"}]}}
           [:key [:ref {:foo "bar"} :my/int]]])
#2023-02-0302:11Noah BogartCan I instrument outside code?#2023-02-0317:25escherizelike in another ns? not currently#2023-02-0303:11Noah BogartLooks like => would have to be modified to use the given symbols namespace, but otherwise works well! Maybe I’ll open a PR#2023-02-0323:10Michael Gardnerwhat's the right way to "unwrap" a :maybe in a map?
[:map [:x [:maybe :int]]] => [:map [:x :int]]
I can't quite figure out how to use malli.util/update(-in) to accomplish this. Do I need to use a walker instead?
#2023-02-0408:30ikitommiJust for one specific field or traversing the schemas and changing all maybe schemas?#2023-02-0414:38ikitommiAll schemas implement the LensSchema , so you can work in same way as with clojure.core + normal data: With clojure core:
(update {:x [:int]} :x get 0)
; => {:x :int}
With Malli:
(mu/update [:map [:x [:maybe :int]]] :x mu/get 0)
; => [:map [:x :int]
when implementing malli.util, there was a quick discussion what if we just implemented the Clojure Core protocols for Schemas, so you could use normal clojure core functions with them. So, this would work:
(update (m/schema [:map [:x [:maybe :int]]]) :x get 0)
; => [:map [:x :int]
… but it would have introduced problems when working with forms (which is just clojure data):
;; correct result when working with a schema
(update (m/schema [:maybe [:map [1 :int]]]) 0 get 1)
; => [:maybe :int]

;; wrong result when applied to form
(update [:maybe [:map [1 :int]]] 0 get 1)
;; => [nil [:map [1 :int]]]
… so decided to add dedicated helpers in malli.util for schemas
#2023-02-0618:16Michael Gardnernice, thank you. I've been wrapping my schemas with m/schema for fail-fast behavior in case of invalid schemas, but it does mean modifying them requires the m.util helpers#2023-02-0414:38ikitommiAll schemas implement the LensSchema , so you can work in same way as with clojure.core + normal data: With clojure core:
(update {:x [:int]} :x get 0)
; => {:x :int}
With Malli:
(mu/update [:map [:x [:maybe :int]]] :x mu/get 0)
; => [:map [:x :int]
when implementing malli.util, there was a quick discussion what if we just implemented the Clojure Core protocols for Schemas, so you could use normal clojure core functions with them. So, this would work:
(update (m/schema [:map [:x [:maybe :int]]]) :x get 0)
; => [:map [:x :int]
… but it would have introduced problems when working with forms (which is just clojure data):
;; correct result when working with a schema
(update (m/schema [:maybe [:map [1 :int]]]) 0 get 1)
; => [:maybe :int]

;; wrong result when applied to form
(update [:maybe [:map [1 :int]]] 0 get 1)
;; => [nil [:map [1 :int]]]
… so decided to add dedicated helpers in malli.util for schemas
#2023-02-0411:29Ben Sless
(defmethod accept :or [_ _ _ _] (into #{} (map accept children))) ;;??
#2023-02-0414:25ikitommiinteresting, would that work with complex types like [:or [:map [:x :int]] [:map [:y :string]]]?#2023-02-0416:52Ben SlessSet of maps?#2023-02-0414:11hanDerPederthis is a bug, right?
(def t (mt/transformer
        {:name :hello
         :decoders {:string (constantly "HELLO")}}))

(def schema-1 [:map [:a :string] [:b :string]])
(def schema-2 [:or [:map [:a :string] [:b :string]]])
(def schema-3 [:or [:map [:a :string]]])
(def schema-4 [:map [:a :string]])

(def v {:a 1234})

(m/decode schema-1 v t) ;; {:a "HELLO"}
(m/decode schema-2 v t) ;; {:a 1234}
(m/decode schema-3 v t) ;; {:a "HELLO"}
(m/decode schema-4 v t) ;; {:a "HELLO"}
#2023-02-0414:22ikitommiIt’s a feature. :or transformer validates the result after transformation, if the result is not valid, it will ignore that branch. This could be improved, e.g. 1. take the first branch which becomes valid after the transformation (current) 2. if none is valid (new) a. take the first branch OR b. take the first branch where the value changed OR c. take the branch is is “most valid” with some value distance metering … none of these would make it bullet proof, but maybe suck less.#2023-02-0414:22ikitommiwhat do you think?#2023-02-0414:22ikitommiI think the 2a or 2b would be simple and good improvements.#2023-02-0414:23ikitommi.. and both would yield same result in your example.#2023-02-0414:33hanDerPederaha, I see.. hmm, tricky.. my case requires me to have multiple transformation passes (default values, some type coercions, etc..) before it's valid. of the cases you outlined 2b would get me closest, but I still think it would fail. previously I tried using a multi-schema instead of or since all objects have an identity key, but ran into the issue mentioned in the FIXME comment; dispatch value is not valid until transformed#2023-02-0414:41ikitommithere is an example for the dispatch key thing with :multi on README: :dispatch values should be decoded before actual values:
(m/decode
  [:multi {:dispatch :type
           :decode/string #(update % :type keyword)}
   [:sized [:map [:type [:= :sized]] [:size int?]]]
   [:human [:map [:type [:= :human]] [:name string?] [:address [:map [:country keyword?]]]]]]
  {:type "human"
   :name "Tiina"
   :age "98"
   :address {:country "finland"
             :street "this is an extra key"}}
  (mt/transformer mt/strip-extra-keys-transformer mt/string-transformer))
;{:type :human
; :name "Tiina"
; :address {:country :finland}}
#2023-02-0414:48hanDerPederRegarding or. Couldnt you merge all the cases and transform using that?#2023-02-0414:53hanDerPederby no means bulletproof either, but fits with the best effort mantra of transformers#2023-02-0414:55ikitommi> Couldnt you merge all the cases and transform using that? > could you describe how this would work?#2023-02-0414:59hanDerPedergiven or schema ?schema, do (reduce mu/merge nil (m/children ?schema)) and call transform on that#2023-02-0415:00hanDerPederand you wouldnt validate the result#2023-02-0415:02hanDerPederi do this when generating datomic pull patterns for or schemas, so i end up overfetching and clean up using strip-keys#2023-02-0509:55ikitommimu/merge overrides if the values are of different types (like normal merge):
(->> [:or
      [:map [:x :int]]
      [:vector :int]
      :int]
     (m/children)
     (reduce mu/merge nil))
; => :int
#2023-02-0509:56ikitommieven if they are maps, I don’t think this would work:
(->> [:or
      [:map [:x :int]]
      [:map [:x :string]]
      [:map [:x [:maybe :uuid]]]]
     (m/children)
     (reduce mu/merge nil))
; => [:map [:x [:maybe :uuid]]]
#2023-02-0509:57ikitommie.g. if someone passes {:x "1"} it fails on the last branch, being invalid.#2023-02-0509:57ikitommiusing me/union might be ok here:
(->> [:or
      [:map [:x :int]]
      [:map [:x :string]]
      [:map [:x [:maybe :uuid]]]]
     (m/children)
     (reduce mu/union nil))
; => [:map [:x [:or [:or :int :string] [:maybe :uuid]]]]
#2023-02-0509:59ikitommiwould work at least in your example:
(->> [:or [:map [:a :string] [:b :string]]]
     (m/children)
     (reduce mu/union nil))
; => [:map [:a :string] [:b :string]]
#2023-02-0510:02ikitommihere, the order matters:
(map
 (m/decoder
  (->> [:or
        [:map [:x :int]]
        [:map [:x :string]]
        [:map [:x [:maybe :uuid]]]]
       (m/children)
       (reduce mu/union nil))
  (mt/string-transformer))
 [{:x 1} {:x "1"}, {:x "kikka"}, {:x (str (random-uuid))}])
;({:x 1} 
; {:x 1} 
; {:x "kikka"} 
; {:x "ed4e918e-2e4f-49d0-a0ee-d9805ffe6384"}) ;; already ok at the string branch
#2023-02-0510:03ikitommiswapping the order:
(map
 (m/decoder
  (->> [:or
        [:map [:x [:maybe :uuid]]]
        [:map [:x :string]]
        [:map [:x :int]]]
       (m/children)
       (reduce mu/union nil))
  (mt/string-transformer))
 [{:x 1} {:x "1"}, {:x "kikka"}, {:x (str (random-uuid))}])
;({:x 1}
; {:x 1}
; {:x "kikka"}
; {:x #uuid"ed4e918e-2e4f-49d0-a0ee-d9805ffe6384"})
#2023-02-0510:36ikitommisummary: 1. mu/union gets you close, but need to be mindful about the order 2. would need more testing to verify that it doesn’t give wrong answers, e.g. with nested values 3. making default transformers depend on malli.utilit would mean than in CLJS, anyone using any transformations, would need to pull a lot of extra code for this, making the js bundle size much larger by default. Should be optional 4. if you think this would solve your case, I would be ok with: a. adding an option to :or to override the default transformation logic with a custom one#2023-02-0612:00hanDerPederThanks for the thorough explanation and outlining the steps going forward. I've gone through our usage of or-schemas. Knowing what I now know about the semantics, I've yet to find a single case where a multi spec isn't preferable. If I find one I'll send a PR.#2023-02-0610:57renewdoitI am trying to transform schemas, but due to the need for documentation, I have to read the source code; one potential problem is there is no API namespace to mark which function/protocol is considered internal and subject to change.#2023-02-0612:52ikitommiGood point. All protocols are part of the extender API. See https://github.com/metosin/malli#alpha, could add that to docs. Not planning on breaking those, will bump a minor if those change. Add a proper description what changed#2023-02-0612:53ikitommithe section also defines the public / extender API difference.#2023-02-0613:50Ben SlessPlaying a bit with persisting schemas in datascript, going by the AST I find its structure is not very regular. A value can be a schema reference or a concrete value. This makes writing a decent schema for when things are components, references and unique a headache. How would you approach this?#2023-02-0614:45refset> This makes writing a decent schema (I assume you mean DataScript schema here) I don't have an answer for you...but I guess it may help to clarify: are you specifically hoping to use Datalog queries to perform analysis on the Malli schema(s)?#2023-02-0615:07Ben Slessyes, specifically, I'm trying to "unify" different schema registries, assuming they specify an ad-hoc key as communication channel, to check for rules and gain insights#2023-02-0615:08Ben Slessfor example, you send a message over a queue and say it's a string which starts with "abc". I read that message and say it's a string. We can unify these constraints#2023-02-0615:08Ben Slessbut to do that with complex schemas I need a way to store them reasonably in a datalog database#2023-02-0616:29refsetCool, that sounds pretty interesting! If you don't have a massive graph, I wonder whether you may have an easier time with using (e.g.) Meander directly on the Malli schemas ...although I've not actually used Meander before, I'm just speculating 🙂#2023-02-0616:42Ben SlessRecursive unification with meander is hard#2023-02-0616:43Ben SlessSpeaking from experience#2023-02-0617:01Ben SlessI do think this is a datalog shaped problem#2023-02-0618:23sun-oneFirst off amazing library, it's been a pleasure to use. My question is related to massive schemas. How do you store and use them? Currently I'm generating malli schemas off of a standard called FHIR which has massive amounts of data describing the datas schemas here's a sample of the size https://gitlab.com/genfhi/genfhi/-/blob/malli_changes/cljc/generator/resources/structure_definitions/structure_definitions/hl7/profiles-resources.json 30+mb json file that defines schema for various data types. I generated a massive clj file https://gitlab.com/genfhi/genfhi/-/blob/schema_alts/cljc/generator/src/gen_fhi/generator/r4/schemas.cljc but unsurprisingly it gets killed on AOT compilation with method size too large (note this works just fails during AOT compilation). Should I store these massive schemas on edn file and read-string them in (related to that I have another issue that comes up in that I have functions that would need to be evaled so I need to read the edn than eval it). Is this the right approach to be generating a giant map of schemas (all these data types ref one another).#2023-02-0618:26sun-oneI could split these into chunks of data that get merged at the end too. But just curious to hear if this is the right approach for schemas of this size.#2023-02-0719:19ikitommithanks! I get "...structure_definitions/hl7/profiles-resources.json" did not exist on "master"#2023-02-0719:20ikitommiIf you could share the files, happy to take a look what can be done#2023-02-0819:42sun-oneOh sorry I moved the branch let me update those links#2023-02-0819:49sun-one@U055NJ5CC links are updated here's the json file representing the source of schemas and my generated malli schema map. https://gitlab.com/genfhi/genfhi/-/blob/malli_changes/cljc/generator/resources/structure_definitions/structure_definitions/hl7/profiles-resources.json https://gitlab.com/genfhi/genfhi/-/blob/schema_alts/cljc/generator/src/gen_fhi/generator/r4/schemas.cljc I suspect I should be reading it from edn and doing the neccessary wiring with sci (I do use dynamic bindings which I'm assuming wouldn't be an issue). Validation was working for me when I run it on dev it's really just the AOT compilation that was causing me issues (I could maybe hack around this and do a trick to avoid AOT comp for this massive map).#2023-02-0622:04escherizeDo time schemas have generators? I couldn’t find anything on that#2023-02-0623:36escherizeMy colleague found:
'[malli.experimental.time :as mtime]
 '[malli.experimental.time.generator :as timegen]
#2023-02-0705:22Ben SlessIs it missing from the documentation?#2023-02-0716:24escherizeProbably not#2023-02-0716:25escherizeIn fact: https://github.com/metosin/malli#generators---malliexperimentaltimegenerator it’s right here#2023-02-0623:36escherizeMy colleague found:
'[malli.experimental.time :as mtime]
 '[malli.experimental.time.generator :as timegen]
#2023-02-0705:59renewdoitm/entries wraps the entries into [::m/varl :int] like reified structure, how do I unwrap the entry get that :int out?#2023-02-0706:06renewdoitAnswer by my own: m/children can access it.#2023-02-0717:37marrsI'm trying to create a schema for a map that can contain one of a number of keys, e.g. a login form schema that can accept :username or :email. I'm sure it must be possible to create a schema to represent this but I can't find mention of it in the docs. I want to write something like
(def schema
  [:map
   [:or
    [:username [:string]]
    [:email [:string]]
   [:password [:string]]])
#2023-02-0718:36Noah BogartCould use an :fn schema:
(def my-schema
  [:and
   [:map
    [:x int?]
    [:y int?]]
   [:fn (fn [{:keys [x y]}] (or x y))]])
#2023-02-0718:43marrsThanks, I'll think about that#2023-02-0718:44escherizemulti schema?#2023-02-0718:45escherizenah, that’s not quite right ^#2023-02-0718:46escherizeI’d go with @UEENNMX0T’s fn approach#2023-02-0719:15ikitommiYes, there is no declarative syntax for dependent keys. One could cook up that with :multi but would be quite ugly. This could be nice (and doable in the user space):
(def schema
  [:and
   [:map
    [:username {:optional true} [:string]]
    [:email {:optional true} [:string]]
    [:password [:string]]]
   [:keys/xor [:username :email]]])
#2023-02-0719:16ikitommibut, not sure if that is worth it. :fn is quite clear already.#2023-02-0719:45escherizekind of rhymes with the https://github.com/bsless/malli-keys-relations library#2023-02-0720:35ikitommiyes, tried to find that as an example here, didn’t, thanks!#2023-02-0809:33marrsThanks, everyone. I ended up making 2 different schemas and choosing between them at runtime depending on which field was presented in the submitted form.#2023-02-0809:45marrs> Yes, there is no declarative syntax for dependent keys. One could cook up that with :multi but would be quite ugly. This could be nice (and doable in the user space): >
(def schema
>   [:and
>    [:map
>     [:username {:optional true} [:string]]
>     [:email {:optional true} [:string]]
>     [:password [:string]]]
>    [:keys/xor [:username :email]]])
> The only thing about this (and its equivalent :fn version) is what would happen if neither :username nor :email were submitted. Both fields are declared as optional which implies that the form should validate, but that's not correct. Each field is required if the other isn't present.
#2023-02-0723:16PanelHi, I hack something up to convert malli schema to EQL querry, it support recursive query. Here's the code with an example at the bottom that convert the burger schema from http://malli.io
(ns malli-eql
  (:require
   [malli.core :as m]
   [malli.util :as mu]))

(defn trim-branch-path [paths]
  (reduce (fn [acc item]
            (if (some #{item} (map #(vec (take (count item) %)) acc))
              acc
              (conj (vec (remove #(#{(butlast item)} %)
                                 acc))
                    item)))
          []
          paths))

(defn -collect [schema]
  (let [state (atom {})]
    (m/walk
     schema
     (fn [schema _ _ _]
       (let [properties (m/properties schema)]
         (doseq [[k v] (-> (m/-properties-and-options properties (m/options schema) identity) first :registry)]
           (swap! state assoc-in [:registry k] v))
         (swap! state assoc :schema schema)))
     {::m/walk-schema-refs true})
    @state))

(defn map-vec-tree-seq [form]
  (tree-seq #(or (sequential? %)
                 (associative? %))
            (fn [form]
              (cond (sequential? form) (next form)
                    (associative? form) (#(interleave (keys %) (vals %)) form)))
            form))
(defn recursive-registry [schema]
  (->> (-> schema m/schema -collect :registry)
       (map (fn [[k v]]
              [k
               (->> v
                    mu/subschemas
                    trim-branch-path
                    (map #(if (and (= (m/type (:schema %)) :ref)
                                   (= k (first (m/children (:schema %)))))
                            (assoc % :in (conj (:in %) '...))
                            %))
                    (map :in))]))
       (filter (fn [[_k v]] (some #{'...} (map-vec-tree-seq v))))
       (into {})))

(defn path->eql [path]
  (let [p' (->> path
                (remove #{:malli.core/in}))

        eql (reduce #(hash-map %2 %1) (if (= '... (last p')) (last p') [(last p')]) (reverse (drop-last p')))]
    (or (if (vector? eql)
          (first eql)
          eql) [])))

(defn deep-merge-eql [forms]
  (reduce (fn f'
            ([] nil)
            ([a b]
                  (let [s (when (and (map? b) (sequential? a))
                            (some
                             (fn [x'] (when (get x' (ffirst b))
                                        x'))
                             a))
                        r (cond (and (vector? a)
                                     (vector? b))
                                (vec (into #{} (concat a b)))

                                s (conj (vec (remove #{s} a)) (deep-merge-eql [s b]))
                                (and (vector? a)
                                     (map? b)
                                     (some #{(ffirst b)} a)) (conj (vec (remove #{(ffirst b)} a)) b)
                                (and (vector? a)
                                     (map? b)) (conj a b)
                                (and (map? a)
                                     (map? b)
                                     (= (ffirst a) (ffirst b))) (merge-with f' a b)
                                (and (map? a)
                                     (map? b)) [a b]
                                (and (vector? b)
                                     (map? a)
                                     (some #{(ffirst a)} b)) (conj (vec (remove #{(ffirst a)} b)) a)
                                (and (map? a)
                                     (vector? b)) (conj b a)
                                (vector? a) (vec (into #{} (conj a b)))
                                (vector? b) (vec (into #{} (conj b a)))
                                :else
                                (vector a b))]
                    r)))
          forms))

(defn schema->paths [schema registry]
  (let [paths (atom [])
        collect-paths-walker! (fn walker'
          ([s] (walker' s []))
          ([s parents]
           (m/walk s (fn [schema path _children _options]
                       (let [is-ref (and (m/-ref-schema? schema)
                                         (m/-ref schema))
                             ref-name (when is-ref (m/-ref schema))
                             in (mu/path->in (m/schema s) path)]
                         (cond (and is-ref
                                    (some #{ref-name} (keys registry)))
                               (mapv #(swap! paths
                                             conj
                                             (vec (concat parents in %)))
                                     (get registry ref-name))

                               (and is-ref ref-name)
                               (walker' (m/deref schema) (vec (concat parents (mu/path->in (m/schema s) path))))
                               :else (swap! paths conj (vec (concat parents in)))))))))]
    (collect-paths-walker! schema)
    @paths))

(defn schema->eql [schema]
  (->> (schema->paths schema (recursive-registry schema))
       trim-branch-path
       (map path->eql)
       deep-merge-eql))

(schema->eql [:schema
              {:registry {"Country" [:map
                                     {:closed true}
                                     [:name [:enum :FI :PO]]
                                     [:neighbors
                                      {:optional true}
                                      [:vector [:ref "Country"]]]],
                          "Burger" [:map
                                    [:name string?]
                                    [:description {:optional true} string?]
                                    [:origin [:maybe "Country"]]
                                    [:price pos-int?]],
                          "OrderLine" [:map
                                       {:closed true}
                                       [:burger "Burger"]
                                       [:amount int?]],
                          "Order" [:map
                                   {:closed true}
                                   [:lines [:vector "OrderLine"]]
                                   [:delivery
                                    [:map
                                     {:closed true}
                                     [:delivered boolean?]
                                     [:address
                                      [:map
                                       [:street string?]
                                       [:zip int?]
                                       [:country "Country"]]]]]]}}
              "Order"])
;; => [{:lines
;;      [:amount
;;       {:burger [:description :name {:origin [:name {:neighbors ...}]} :price]}]}
;;     {:delivery
;;      [:delivered {:address [:street :zip {:country [:name {:neighbors ...}]}]}]}]
#2023-02-0923:32JoelWow this is eery, I came here to ask a question about building some pathom code from malli. Are you working on something open-source and using pathom?#2023-02-0923:38PanelI’m also playing with pathom and malli, nothing worth publishing atm. But I think it might be useful to have more tool to transform malli schema into other form to be used to generate form/pathom/db schema/… The actual part relevant to eql in the above code is not much.#2023-02-0923:40Joeli had written code to pull data from a json request using malli, which i realized is too naiive (like nested data). But, also wanting to pass that to pathom to query as well.#2023-02-0923:42Joeli’m putting a “pathom like attribute” in the properties sections in malli schema to do the mapping, but the ::pco/output for pathom needs a more accurate mapping.#2023-02-0923:43Joelis that what you are generating above?#2023-02-0923:49PanelThe code is generating the query, but it could be adapted to generate pathom output form easily. But those would be dependent on the underlying data store, so for example the above burger example could be split into many different resolver.#2023-02-1000:36Michael Gardneris there an established "best practice" re: calling m/schema or not in your schema defs? E.g.
(def foo-schema
  [:map ...])
vs
(def foo-schema
  (m/schema
   [:map ...]))
So far I've been writing the latter to get fail-fast behavior when a schema is invalid, but it's more verbose and forces the use of malli.util fns to transform/combine schemas. And AFAIK there's only a small performance benefit to calling m/schema thanks to Malli's caching. So I'm not sure what to pick as a default.
#2023-02-1001:42rutledgepaulvOne suggestion could be to write a macro which defs and validates a schema but defines the value of the var as the original data as long as validation succeeds#2023-02-1001:43rutledgepaulvOr a function that does the same and just use that instead of m/schema in your second example#2023-02-1018:22Michael GardnerI could certainly do that, although it's still a little more verbose. And I'm not even sure I need it given that my data schemas will end up getting validated at ns load time via the function schemas that use them anyway. So I guess my only remaining question is about performance. Is it true that there's no meaningful performance benefit to calling m/schema any more?#2023-02-1114:40Varghiz cljHi, We've a use case where we need to apply schema validation on the body based on the type value in the header. For example if the type is foo then should apply schema/foo . If the type is bar then should apply schema/bar.. dynamically#2023-02-1114:49Ben SlessYou can always add a middleware which adds it to the body from header before coercion#2023-02-1114:55Varghiz cljThank you @UK0810AQ2..I'm new to malli & reitit, can you please give me a reference or sample on middleware#2023-02-1114:55Ben SlessYou're using malli in http, are you using it with reitit?#2023-02-1114:57Varghiz cljI use malli for schema validation and reitit.swagger for API definition and routes.#2023-02-1115:06Ben SlessHow does your ring handler look like?#2023-02-1115:15Varghiz clj
["/dosomething"
    {:swagger {:tags ["do something"]}
     :post    {:summary    "route the call after validating the payload based on foo or bar"
               :scope      [:api:write]
               :parameters {:header [:map
                                     [:authorization schema/authorization]
                                     [:type schema/type-header]]          ;; foo or bar
                            :body [:map
                                   [:events schema/general-body]]}        ;; this is the part we need to call schema/foo or schema/bar based on type
               :handler    (h/process-api config)}}]
#2023-02-1115:15Varghiz cljthe part events is a generic such that based on the type, we will apply one schema vs the other#2023-02-1115:18Ben SlessThat's not the ring handler, just reitit routes#2023-02-1115:19Varghiz cljoops.. sorry#2023-02-1115:24Varghiz clj
(defn process-api [config]
  (fn [{{:keys [header body]} :parameters}]
    (let [events (body :events)
          total  (count events)]
      (let [results    (k/send-events config events header)
            successful (count (remove :error results))]
        {:status (cond
                   (zero? successful) 400
                   (= total successful) 200
                   :else 206)
         :body   {:events results}}))))
#2023-02-1116:20Ben SlessYou should share your entire code, what about reitit.ring/ring-handler#2023-02-1221:41steveb8nQ: should a uuid decoded by transit satisfy the :uuid built-in schema? I found it does not on cljs. Bug or feature?